Cloudflare Crawl-to-Referral Ratios: A Defensible Audit Method

Direct answer: Cloudflare’s crawl-to-referral ratio compares attributed HTML crawl requests with attributed HTML referral requests for a platform. It can reveal operational imbalance. It cannot prove model training, answer use, citation, causation, fairness, revenue, or the value of a crawler.

Two dated product surfaces must not be collapsed. Cloudflare Radar introduced aggregated crawl-and-refer insights and the crawl_refer_ratio dimension on July 1, 2025. On July 1, 2026, Cloudflare announced BotBase and Attribution Business Insights for Enterprise Bot Management, including site-wide and per-operator ratios alongside bot traffic. The first is aggregated Radar context; the second is a customer dashboard for eligible enterprise zones.

The decision this guide helps you make

Use this method before changing bot rules because a ratio looks high, before presenting a crawler as “unfair,” or before claiming that crawling produced no return. The immediate task is to reconstruct numerator, denominator, time window, scope, classification, and missing-referrer behavior. Policy comes after measurement quality and business context.

Keep the 2025 and 2026 releases separate

This chronology corrects the ambiguity in the earlier version of this article. The metric concept predates the enterprise dashboard by one year.

Reconstruct the formula

Cloudflare’s Radar glossary says crawl requests for HTML pages are mapped from the User-Agent header and referral requests for HTML pages from the Referer header, by platform. Express the operational ratio as:

crawl-to-referral ratio = attributed HTML crawl requests / attributed HTML referral requests

Record whether the displayed product computes the ratio for you or whether you recomputed it from exported counts. Preserve the exact metric name, product surface, account scope, and retrieval date.

Handle zero and near-zero denominators explicitly

A ratio can become extremely large because referrals are close to zero even when crawl volume is ordinary. Never publish the ratio without both raw counts. If referrals are zero, report “undefined” or a declared zero-denominator state—not infinity, zero, or a dropped row. Avoid averaging ratios across time buckets unless the weighting method is stated.

These are labelled examples, not Cloudflare or site results. The downloadable crawl-referral interpretation ledger marks every sample row EXAMPLE-REMOVE.

Treat missing native-app referrers as a known limitation

Cloudflare explicitly notes that traffic from native apps may omit the Referer header. Because Radar referral counts include web-based tools, the calculated ratios may be overstated by an unknown amount. This is not a minor footnote: it directly affects the denominator.

Do not invent a correction factor. Instead, name the limitation, inspect tagged destination URLs and first-party analytics where lawful, and report the additional observations separately. Absence of a standard referrer does not prove absence of exposure or visits.

Compare windows without mixing them

A 24-hour window can be dominated by a crawl burst or incident. Seven- and 30-day windows can absorb product releases, content launches, seasonality, or rule changes. Store a time series and annotate changes rather than relying on one screenshot. The aggregation resolution can vary with the selected Radar timeframe, so record it.

When comparing periods, keep the same zone scope, operator definition, action state, and response classes. A changed filter can look like a changed crawler.

Segment the site before drawing policy conclusions

A site-wide ratio can hide a heavily crawled archive, a rarely crawled commercial section, and a blocked asset path. Where data permits, segment by hostname, path family, content type, status class, bytes, and action. Track HTML pages separately from resources that are outside the documented ratio.

Separate operator attribution from request verification

A platform label is an observation layer. User-agent strings can be spoofed. Cloudflare’s verified-bot documentation and Web Bot Auth describe stronger verification contexts, but a “verified” label is not a moral or business-value judgment. Record the detection source, verified status where available, bot category, operator, and a sample of raw request evidence.

Use the AI bot access policy framework to connect classification to a documented decision. Do not change policy solely because an operator name appears in a chart.

Reconcile three evidence layers without forcing agreement

Keep edge-request evidence, Cloudflare’s classified product view, and first-party analytics as separate but joinable records. Edge logs answer which requests reached the property and how the site responded. Cloudflare’s bot layer supplies the operator, category, verification, and action labels available in the selected product. Analytics supplies sessions and declared outcomes under its own consent, attribution, filtering, and referrer rules.

The totals may disagree for legitimate reasons: timezones, bot filtering, redirects, client-side collection, consent, URL tagging, native-app referrer loss, or a different definition of an HTML request. Do not “fix” the evidence by copying one system’s count into another. Record the coverage difference, investigate material gaps, and state which source governs each decision.

A policy owner may use verified edge volume to manage infrastructure while a commercial analyst uses tagged sessions and on-site outcomes to evaluate observable value. Neither layer proves answer-level citation or training use. If the organization needs those answers, commission separate studies with their own sampling frames and review criteria.

What the metric cannot prove

• That a fetched page trained a model.
• That a page was retrieved for a particular answer.
• That an answer cited, paraphrased, or absorbed the page.
• That crawl volume caused referral volume.
• That a crawler is beneficial, harmful, fair, or unfair.
• That a block would create leverage, licensing revenue, or more visits.
• That low recorded referrals mean zero brand exposure.

To study answer use, collect prompts, environments, answers, citations, dates, and supported claims. To study value, connect observable visits to declared on-site outcomes. Keep those studies linked by dates and operators, but do not merge their evidence into the ratio.

Test a rule change as an intervention

Before altering rules, save a baseline and a rollback point. Declare the target bot or behavior class, affected paths, expected operational effect, guardrails, test window, and decision owner. Concurrent releases, outages, migrations, or content changes belong in the same ledger.

The small SEO experiment method provides a compact intervention design. It does not create a causal result automatically; it helps preserve what changed and what did not.

Use decision rules that preserve uncertainty

• Investigate classification when request samples contradict the assigned operator.
• Report “undefined” for a zero referral denominator.
• Do not compare ratios without raw counts and compatible scopes.
• Do not attribute app-origin traffic loss to user demand without accounting for missing referrers.
• Escalate policy changes only after ownership, rollback, and guardrails are documented.
• Use a no-conclusion state when coverage or classification is insufficient.

Primary documentation and editorial boundary

Cloudflare’s July 1, 2025 Radar changelog documents the crawler endpoints and ratio dimension. The Bot Management changelog documents the July 1, 2026 BotBase and Attribution Business Insights release. The Radar glossary defines the header-based mapping and native-app referrer limitation.

Search Engine Answer has not independently reproduced Cloudflare’s classifications or compared the customer dashboard across a publisher panel. This guide interprets documented fields and supplies an audit method; it does not claim a universal ratio threshold or a causal return estimate.