How much do AI crawlers crawl compared to what they send back?
Cloudflare Radar's 17 November snapshot: 3,351.73 Perplexity crawls, 2,073.96 OpenAI crawls and 456.23 Anthropic crawls per observed referral.
GPTBot, ClaudeBot, OAI-SearchBot and friends: crawl volumes, referrals, robots.txt and blocking.
1 story
Cloudflare Radar's 17 November snapshot: 3,351.73 Perplexity crawls, 2,073.96 OpenAI crawls and 456.23 Anthropic crawls per observed referral.