AzokleBot / 2.0
AzokleBot is the automated web crawler operated by Azokle. It crawls public web pages, images, videos, and news to power Azokle Search and ground Azokle AI.
Official User-Agent
AzokleBot exclusively identifies itself with the following HTTP request header:
Public Web Data Indexed
AzokleBot indexes four key public content categories.
Web Pages & Articles
Discovers public websites, documentation, and technical articles for unbiased search results and factual AI grounding.
Images & Visual Media
Extracts image metadata, resolutions, alt descriptions, and licensing tags for Azokle Image Search.
Videos & Transcripts
Parses video titles, timestamps, thumbnails, and public transcripts for fast multimedia discovery.
News & Publications
High-frequency discovery of live journalism feeds and breaking chronological news updates.
Robots.txt Directives
Select your policy below to generate rules for your /robots.txt file.
# Allow AzokleBot to index public content for Azokle Search & AI User-agent: AzokleBot Allow: / Sitemap: https://example.com/sitemap.xml
<meta name="AzokleBot" content="noindex, nofollow">Prevents specific web pages from appearing in Azokle Search.
X-Robots-Tag: noindex, noarchiveRestricts indexing for non-HTML assets like PDF documents and media.
Verify Legitimate AzokleBot Traffic
Because User-Agent headers can be spoofed by third parties, use standard Reverse DNS verification on accessing IP addresses:
*.crawl.azokle.com or *.bot.azokle.com
HTTP Response Handling
Processed and queued for search indexing.
Follows redirect and updates canonical URL.
De-indexes the URL from search results.
Backs off request rate automatically.