Why it matters
- Cloudflare Radar’s 17 November 2026 snapshot records 3,351.73 Perplexity crawls, 2,073.96 OpenAI crawls and 456.23 Anthropic crawls per observed referral over the last seven days.[7][1]
- The OpenAI and Anthropic figures are platform-level ratios, not GPTBot-only or ClaudeBot-only measurements.[3]
- Fastly measured approximately 30% AI-request growth from January through May 2026, around 6.5 times the growth rate of human traffic.[11]
- Cloudflare Pay Per Crawl remains documented as a closed-beta payment mechanism, with no cited market-wide transaction or payout totals.[17][15]
- Cloudflare’s November 2026 access-policy documentation lists Allow as the new-domain default for Search, Agents and Training, so older default-blocking headlines need a date and qualification.[26][13]
As of 17 November 2026, AI platforms still crawl substantially more HTML than the measurable referrals they send to publishers, but the gap varies by platform and observation window: Cloudflare Radar’s latest seven-day snapshot records about 2,073.96 OpenAI crawls per observed referral, 456.23 Anthropic crawls, and 3,351.73 Perplexity crawls.[1][2] These are platform-level figures, not ratios for GPTBot or ClaudeBot alone, and Cloudflare warns that missing referral headers—particularly from native apps—can inflate the measured gap.[1]
What happened?
Cloudflare Radar’s 17 November 2026 snapshot shows a large but uneven difference between HTML crawling and observed referrals. Perplexity recorded the highest current platform-level ratio at 3,351.73 crawls per observed referral, followed by OpenAI at 2,073.96 and Anthropic at 456.23.[3][1]
The measurement covers the dashboard’s last seven days, rather than the full month of November.[1] Cloudflare’s calculation counts HTML crawl requests from associated crawler user agents and compares them with HTML requests containing a platform-associated Referer hostname.[4]
That methodology means the figures should be described as crawls per observed referral, not crawls per visitor, customer, conversion or unit of publisher revenue.[1] They also do not establish how many requests came specifically from GPTBot or ClaudeBot, because the platform aggregates multiple bots under each operator.[4][1]
Cloudflare’s rendered bot-traffic chart separately displays ClaudeBot at 12.1% and GPTBot at 8.1%, but the accessible chart text does not define whether those percentages use all HTTP requests or a crawler-only denominator.[5][6] They should therefore be reported as the chart’s displayed percentages, not as shares of the entire web.[7]
Why does the crawl-to-referral gap matter?
The gap matters because crawling and clicking produce different outcomes for publishers. A crawler can retrieve a page for training, search indexing, answer generation or an agent task without sending a person through to the publisher’s site.[8][9]
The current measurements do not show that every crawl is intended for model training. Cloudflare reported that training represented 52% of crawler requests in June 2026, compared with 22% in spring 2025, while its classification also distinguishes search, agent and other purposes.[8]
Fastly’s 2026 data shows that AI traffic is growing as an infrastructure workload as well as a content-distribution issue. In a fixed customer cohort covering January 1 through May 31, 2026, AI requests grew by approximately 30%, around 6.5 times faster than human traffic.[10][11]
Fastly also reported that AI requests were more likely than human requests to reach origin infrastructure: less than 9% of human requests reached origin in May 2026, compared with more than 51% of AI requests.[11] That difference affects caching, bandwidth, origin capacity and bot-management decisions, although the cited analysis does not provide a dollar cost per request.[8]
The economic question is separate from the technical one. A 2026 academic study using U.S. desktop clickstream data from October 2024 through July 2025 found a clean outbound click in 5.2% of ChatGPT conversation sessions, compared with 31.1% of Google queries.[2] Those are user-behaviour measurements, not crawler-to-referral ratios, and neither result measures publisher revenue.[12][1]
What do the latest measurements show?
Cloudflare’s current platform-level values are:
| Crawler or platform | Operator | Share or ratio | Source | Date |
|---|---|---|---|---|
| GPTBot and related OpenAI bots | OpenAI | 2,073.96 crawls per observed referral | Cloudflare Radar | 17 November 2026[1] |
| ClaudeBot and related Anthropic bots | Anthropic | 456.23 crawls per observed referral | Cloudflare Radar | 17 November 2026[1] |
| Perplexity crawler group | Perplexity | 3,351.73 crawls per observed referral | Cloudflare Radar | 17 November 2026[12][5] |
| ClaudeBot | Anthropic | 12.1% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |
| GPTBot | OpenAI | 8.1% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |
| Googlebot | 25.5% of the chart’s displayed bot-traffic measure | Cloudflare Radar | 17 November 2026[13][1] |
Cloudflare’s current ratio differs sharply from its historical 2025 figures. In the monthly comparison for January and July 2025, Anthropic fell from 286,930.1:1 to 38,065.7:1, OpenAI moved from 1,217.4:1 to 1,091.4:1, and Perplexity rose from 54.6:1 to 194.8:1.[1]
Those monthly values cannot be treated as a direct continuation of the November 2026 rolling-week snapshot. The observation windows, measurement periods and platform activity changed between the datasets.[4][14]
Cloudflare’s 2025 historical analysis also recorded different movements among individual bots. Between May 2024 and May 2025, GPTBot requests grew by 305%, while ClaudeBot requests fell by 46% in the fixed-customer comparison.[13][1]
The data therefore does not support one universal ranking such as “the worst crawler” or a permanent GPTBot crawl ratio. The answer depends on the operator, included user agents, time window, referral definition and traffic classification.[2]
What is confirmed—and what is not?
Confirmed
- Cloudflare’s 17 November 2026 Radar snapshot reports platform-level crawl-to-observed-referral ratios of 3,351.73:1 for Perplexity, 2,073.96:1 for OpenAI and 456.23:1 for Anthropic.[1]
- The snapshot covers the dashboard’s last seven days, not a full-November average.[7]
- OpenAI and Anthropic platform totals aggregate multiple user agents, so the ratios cannot be assigned to GPTBot or ClaudeBot alone.[4]
- Fastly measured approximately 30% AI-request growth from January through May 2026, with AI traffic growing about 6.5 times faster than human traffic.[11][10]
- Cloudflare’s Pay Per Crawl documentation still describes the product as closed beta in documentation updated 28 July 2026.[15]
- Cloudflare’s access-policy documentation updated 2 November 2026 lists Allow as the new-domain default for Search, Agents and Training.[16][13]
Not confirmed by the cited evidence
- Absolute November 2026 request totals for GPTBot or ClaudeBot are not published in the cited Cloudflare dashboard responses.[1]
- The cited sources do not publish a universal AI-crawler-to-revenue ratio.[17]
- Pay Per Crawl’s cited documentation does not publish aggregate paid-request volume, publisher payouts or the number of paying AI operators.[8][1]
- The cited material does not provide a general-availability date for Pay Per Crawl.[17][15]
- Fastly’s reported Claude growth is not identified as ClaudeBot-only traffic.[10]
- Akamai’s 2026 commerce findings cover July through December 2025, while the 2026 Thales/Imperva report analyses full-year 2025.[18]
How are pay-per-crawl and blocking changing?
Cloudflare introduced Pay Per Crawl in private beta on 1 July 2025. The system allows site owners to select crawlers to charge, set prices, manage payments through Stripe and monitor successful content deliveries.[19]
The documentation updated on 28 July 2026 describes a payment negotiation using HTTP headers. A crawler can present payment intent, or receive an HTTP 402 Payment Required response containing pricing information; successful payment can permit an HTTP 200 response.[15]
The system charges again when the same page is crawled again, while errors are not billed. Paths such as /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json remain free.[20]
Cloudflare expanded the configuration on 16 June 2026 with URI-pattern exclusions and dynamic pricing. Publishers can leave selected pages free while charging for others, and can set prices through an origin response header or a Cloudflare Worker.[21]
This is a documented technical mechanism, not evidence that the AI-crawling market has broadly adopted paid access. The cited product material does not quantify transactions, payments or publisher income.
Blocking controls have also changed over time. Cloudflare’s 1 July 2025 announcement introduced permission-based onboarding for new domains and said more than one million customers had already selected an earlier one-click blocking option.[22]
A later Cloudflare announcement dated 1 July 2026 described Search, Agent and Training controls for all customers and stated that new domains were scheduled to block Training and Agent traffic on ad-displaying pages from 15 September while allowing Search.[23]
However, the Cloudflare documentation updated on 2 November 2026 lists Allow as the new-domain default for Search, Agents and Training.[16][13] The cited documentation does not explain the difference between the announcement and the later setting.
Current controls distinguish Block, Block on pages with ads, Allow and Disallow. Cloudflare describes Disallow as a Training-only instruction conveyed through robots.txt, which remains voluntary unless additional edge enforcement is enabled.[16]
With Bot Preference Sync enabled, Cloudflare can enforce Disallow at the edge for training crawlers, but mixed-purpose Search-and-Training crawlers are excluded from that edge block.[16] Cloudflare separately reported in September 2026 that fewer than 1% of its sites blocked search crawlers, compared with 17% blocking training.[8]
What do other 2026 measurements add?
Fastly’s January-May 2026 analysis found that AI requests were composed of approximately 85% crawlers and 15% fetchers in May.[10][11] Fastly defines fetchers as traffic associated with user-directed requests, while crawlers represent more scheduled or automated activity.[10][11]
Fastly also reported Claude-related traffic at more than 555% above its January baseline during the analysis period.[11] The source calls this “Claude traffic” and does not establish that the figure represents ClaudeBot alone.[4][3]
A separate Fastly investigation examined 23 September 2026 logs and found approximately 6.3 million requests from nearly 10,200 IP addresses falsely claiming AI-crawler or fetcher identities.[24] The finding shows why a user-agent string alone cannot establish that a request came from the claimed operator.[1][24]
Akamai’s publishing and commerce measurements cover a different population. Its publishing report uses application-layer traffic observed through Akamai bot-management tools during 1 July-31 December 2025.[17][18]
Akamai reported that commerce represented approximately 48% of AI-bot activity and media represented 13% in the relevant dataset.[17] Its July 2026 commerce announcement gave a more precise commerce figure of 47.9% and said training crawlers accounted for more than 70% of AI-bot triggers in commerce during that period.[18]
Imperva’s 2026 Thales Bad Bot Report analyses full-year 2025 activity and reported that bots made up more than 53% of web traffic while AI-driven bot attacks increased 12.5 times.[25][9] The attack statistic should not be presented as legitimate GPTBot or ClaudeBot growth.[6]
When should publishers use these measurements?
Publishers should match the metric to the decision:
- Use Cloudflare’s ratio when comparing observed HTML crawling with observed platform referrals, while stating the seven-day window and referral-header limitation.[4]
- Use Fastly’s figures when discussing AI-request growth, origin exposure, crawler-versus-fetcher composition or impersonated bot traffic.[24][6]
- Use Akamai’s figures for sector-specific AI-bot exposure, not for all-web traffic or crawl-to-referral rankings.[18]
- Use Imperva’s figures for automation and bad-bot context, not as a measure of useful crawler demand.[6]
- Use Cloudflare’s policy documentation when describing current blocking controls and Pay Per Crawl’s documentation when describing the closed-beta payment mechanism.[9]
- Use the academic study when discussing user click behaviour and possible search substitution, not when calculating crawler efficiency.[2]
The central distinction is between requests, crawls, fetches, referrals, clicks, visits and revenue. The cited evidence measures different points in that chain and does not justify collapsing them into one “AI traffic” number.[12][7]
What is the background to GPTBot and ClaudeBot traffic?
An AI crawler may retrieve a page for several purposes, including search discovery, answer generation, user-directed fetching or model training.[8] Operators can also use multiple user agents, which means a platform-level measurement may include more than one bot identity.[4]
Cloudflare’s current ratio groups OpenAI-associated bots and Anthropic-associated bots at the platform level.[7][23] That is why a statement such as “GPTBot crawls 2,074 pages for every visitor” would overstate what the data proves: the published ratio applies to the OpenAI group and to observed referrals under Cloudflare’s methodology.[4]
Referral headers also do not capture every user action. Cloudflare notes that Claude’s native application does not include the relevant Referer header, and similar omissions may occur elsewhere.[4] The resulting ratio can therefore overstate the difference between crawling and referrals by an unknown amount.[4]
Background: what does “crawl-to-referral ratio” mean?
A crawl-to-referral ratio compares qualifying HTML crawl requests with HTML requests carrying a platform-associated referral hostname.[4][3] For example, a ratio of 2,073.96:1 means Cloudflare observed approximately 2,073.96 qualifying OpenAI-associated crawl requests for each qualifying OpenAI-associated referral in the stated measurement window.[4][3]
It does not mean that one platform fetched exactly that many unique pages, that every request reached a publisher’s origin, or that the platform generated no visits outside the observed referral-header method.[24][4] It also does not measure revenue or the value of a visit.[2]
Timeline
| Date | Event | What it establishes |
|---|---|---|
| 1 July 2025 | Cloudflare announces Pay Per Crawl in private beta.[19] | A payment mechanism for selected crawler access was introduced; it was not evidence of market-wide adoption.[12][2] |
| July 2025 | Cloudflare publishes historical crawl-to-click data.[1] | Anthropic, OpenAI and Perplexity showed materially different monthly ratios during 2025.[1] |
| 1 July 2025 | Cloudflare announces permission-based AI-crawler onboarding for new domains.[22][2] | The announcement concerned new-domain onboarding and did not establish automatic blocking for every existing site.[2] |
| 16 June 2026 | Cloudflare adds advanced Pay Per Crawl configuration.[21] | URI exclusions and dynamic pricing become documented capabilities.[21] |
| 9 June 2026 | Fastly reports January-May AI-request growth.[11][10] | AI requests grew approximately 30%, around 6.5 times faster than human traffic.[11] |
| 28 July 2026 | Cloudflare Pay Per Crawl documentation is updated.[15] | The service is still described as closed beta.[15] |
| 23 September 2026 | Fastly measures false AI-bot identities in WAF logs.[24] | User-agent claims alone cannot authenticate crawler origin.[24] |
| 30 September 2026 | Cloudflare reports agent-request and crawler-purpose data.[12] | Daily AI-agent requests had grown more than 1,700% year over year; 52% of June 2026 crawler requests were classified as training.[8] |
| 2 November 2026 | Cloudflare access-policy documentation is updated.[5][13] | New-domain defaults are listed as Allow for Search, Agents and Training.[13] |
| 17 November 2026 | Cloudflare Radar snapshot is retrieved.[7] | The last-seven-day platform ratios are 3,351.73:1 for Perplexity, 2,073.96:1 for OpenAI and 456.23:1 for Anthropic.[1] |
Frequently asked questions
What is the AI crawler traffic ratio in 2026?
In Cloudflare Radar’s 17 November 2026 last-seven-day snapshot, Perplexity recorded 3,351.73 crawls per observed referral, OpenAI recorded 2,073.96, and Anthropic recorded 456.23.[1] These are platform-level ratios based on Cloudflare’s HTML-request and referral-header methodology, not universal measures of visits or revenue.[1]
What is the current GPTBot crawl ratio?
The cited November 2026 Cloudflare data does not publish a GPTBot-only ratio.[1][8] It reports an OpenAI platform-level ratio of 2,073.96 crawls per observed referral, aggregating OpenAI-associated bots rather than isolating GPTBot.[3][1]
Does ClaudeBot send less traffic than GPTBot?
The available data does not support that direct conclusion. Cloudflare’s chart displays ClaudeBot at 12.1% and GPTBot at 8.1% of an unspecified chart denominator, while its platform-level crawl-to-referral ratio is lower for Anthropic than OpenAI.[4][5] Those measures use different groupings and denominators.[12]
Is Cloudflare Pay Per Crawl generally available?
No. Cloudflare documentation updated on 28 July 2026 still describes Pay Per Crawl as closed beta.[15] The cited documentation does not provide a general-availability date or aggregate data on paid requests and publisher payouts.[12][2]
Does Cloudflare block AI crawlers by default in 2026?
The answer depends on the date and control being described. Cloudflare’s documentation updated 2 November 2026 lists Allow as the new-domain default for Search, Agents and Training, while earlier 2025 and 2026 announcements described permission-based or planned blocking policies.[13][16] The dated documentation should be cited instead of using an undated claim that Cloudflare blocks all AI crawlers by default.
Sources
- 1The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals blog.cloudflare.com
- 2Answering Without Referring: How AI Search Rewrites the Web’s Economic Bargain Qiaoni Shi, Kai Zhu, and Kai Gu Bocconi University, Milan, Italy arxiv.org
- 3The 2025 Cloudflare Radar Year in Review- the rise of AI, post-quantum, and record-breaking DDoS attacks blog.cloudflare.com
- 4The crawl before the fall… of referrals: understanding AI’s impact on content providers blog.cloudflare.com
- 5AI Insights | Cloudflare Radar radar.cloudflare.com
- 6BAD BOT REPORT imperva.com
- 7AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence arxiv.org
- 8The Internet has a second audience blog.cloudflare.com
- 9AI Bot Traffic: Which Bots to Allow or Block | Imperva imperva.com
- 10AI Traffic Growth: Bots Outpace Humans | Fastly fastly.com
- 11AI Traffic Grew 6.5x Faster Than Human Traffic This Year, Creating New Business Challenges and Opportunities - Tue, 06/09/2026 - 08:00 investors.fastly.com
- 12The Impact of LLMs on Online News Consumption and Production arxiv.org
- 13From Googlebot to GPTBot: Who’s crawling your site in 2025 blog.cloudflare.com
- 14Akamai Research: AI Bots Threaten the Foundation of Web-Based Business Models | Akamai akamai.com
- 15What is Pay Per Crawl? · Cloudflare AI Crawl Control docs developers.cloudflare.com
- 16AI Crawler and Agent access policies developers.cloudflare.com
- 17AI Bot Defense and Monetization for Publishers | Akamai akamai.com
- 18Akamai Research: Commerce Becomes the Epicenter for AI Bot Attacks and Agentic Fraud in 2026 akamai.com
- 19Introducing Pay Per Crawl (private beta) developers.cloudflare.com
- 20Pay Per Crawl FAQ developers.cloudflare.com
- 21Pay Per Crawl advanced configuration developers.cloudflare.com
- 22Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model | Cloudflare cloudflare.com
- 23New options to manage AI traffic developers.cloudflare.com
- 24AI Bot Cosplay | Fastly fastly.com
- 25AI-driven Bot Attacks Surged 12.5x According to Thales Bad Bot Report - Company imperva.com
- 26Your site, your rules: new AI traffic options for all customers blog.cloudflare.com



