Cloudflare Crawl-to-Referral Ratios: What the Metric Can and Cannot Prove
Cloudflare now exposes crawl-to-referral ratios by operator. The ratio measures request imbalance, not citation, training use, revenue, or causal value.
Confirmed product change: Cloudflare’s July 1, 2026 Bot Management update introduced BotBase and Attribution Business Insights with sitewide and per-operator crawl-to-referral ratios over 24-hour, seven-day, and 30-day windows. Cloudflare also exposes classification and action state.
The metric can show an operational imbalance between HTML crawl requests and HTML referral requests attributed to a platform. It cannot prove that crawled content trained a model, appeared in an answer, earned a citation, caused a visit, or created business value.
Define the ratio before interpreting it
Cloudflare’s Radar documentation describes crawl_refer_ratio as a relationship between HTML page crawl requests and HTML page referrals by platform. A ratio needs its numerator, denominator, window, scope, and classification method.
| Field | What it shows | What it does not show |
|---|---|---|
| Crawl count | Attributed automated HTML requests | How content was later used |
| Referral count | Attributed HTML visits from the platform | All exposure or conversions |
| Ratio | Relative request imbalance | Causal value or fairness |
| Action state | How the site handled the bot | Whether the classification is perfect |
Avoid denominator traps
A ratio can become very large when referrals are close to zero, even if crawl volume is modest. Report the raw counts beside the ratio and define how zero-referral periods are handled. Do not average ratios across periods without weighting and explaining the calculation.
Compare the 24-hour, seven-day, and 30-day windows carefully. A crawler burst can dominate the short window while a product change or seasonal referral pattern shapes the longer one. Preserve time series rather than a single screenshot.
Check site scope too. A sitewide ratio can hide a heavily crawled archive and a rarely crawled commercial section. Segment by hostname, response class, path family, content type, and bytes where the available data permits it.
Separate attribution from verification
Bot classification is an observation layer. User-agent strings can be spoofed, shared infrastructure can complicate identity, and product behavior can change. Record the Cloudflare operator label, verified-bot status where available, rule action, and raw request evidence.
Referral attribution also depends on referrer behavior, browser privacy, applications, redirects, and analytics collection. A platform can expose a source without producing a standard referrer, and a referral can be lost before analytics records it.
Use server logs, Cloudflare data, and analytics as complementary observations. The AI bot access policy framework helps connect classification to a documented allow, block, or rate-limit decision.
What the metric cannot prove
- That the platform used a fetched page in a model response.
- That an answer cited or paraphrased the publisher.
- That crawl volume caused referral volume.
- That one crawler is more valuable than another.
- That blocking would increase bargaining power or revenue.
- That low referrals mean no brand exposure occurred.
To study citation, run an answer-level sample with saved prompts, environments, responses, citations, and dates. To study value, connect observable referrals to on-site outcomes under a declared attribution model. Keep those studies separate from crawl policy.
Publish a defensible report
Include raw crawl and referral counts, ratio formula, time windows, scope, operator classifications, rule actions, missing data, and any site changes. Preserve zero and unknown values. Explain whether ratios were calculated by Cloudflare or recomputed.
SearchEngineAnswer has not independently reproduced Cloudflare’s platform classification or compared the metric across a publisher panel. This article interprets the documented product fields and defines the evidence required for a stronger study.
Use the small SEO experiment method when testing a policy change, and save a baseline before modifying rules.
Ask a question or join the discussion