Open robots.txt census of the Tranco top 1M: which AI crawlers (GPTBot, CCBot, ClaudeBot, ...) each site blocks. CC BY 4.0. https://crawlora.net/ai-crawler-index
-
Updated
Oct 12, 2026
Open robots.txt census of the Tranco top 1M: which AI crawlers (GPTBot, CCBot, ClaudeBot, ...) each site blocks. CC BY 4.0. https://crawlora.net/ai-crawler-index
To associate your repository with the ccbot topic, visit your repo's landing page and select "manage topics."