Cloudflare and OpenAI’s discovery pilot leaves publisher outcomes to measure

Cloudflare and OpenAI report that their discovery pilot brought hundreds of millions of previously undiscovered pages into OpenAI’s search index. For publishers, the next question is whether a newly discovered page becomes a useful citation, a visit, or a completed reader task. The announcement does not establish those outcomes.

What the pilot announcement establishes

The companies’ October 15 report describes a pilot launched in July with a cohort covering roughly 1% of Cloudflare’s network. Cloudflare supplies discovery signals; OpenAI retains its own indexing and ranking decisions. The companies report that crawling can initially increase as previously unknown pages are found. They identify freshness signals and business-outcome measurement as further work.

These are company-reported pilot findings. A discovery signal is separate from permission to fetch a page and from permission to use its content for training. The report does not promise inclusion or traffic for every Cloudflare customer.

Follow a page through four separate evidence stages

A publisher can make this announcement useful by defining the next observation before changing an access rule. The following worksheet is a proposed audit method. SearchEngineAnswer has not obtained pilot-level logs or measured a referral lift.

Resolve the first missing observation

Consider a fictional documentation publisher whose request logs show a newly crawled troubleshooting page. The team has no captured answer linking to it and no identified referral arrival. The useful next step is to inspect whether the page appears in a bounded set of relevant answers. Increasing a traffic forecast at that point would skip two unobserved stages.

A different fictional case has a captured citation and a completed visit, but the visitor leaves before finding a download. That calls for checking whether the destination satisfies the question. It does not establish that discovery failed. Keep the actual evidence with each case so a dashboard label cannot silently turn a request into a reader.

1. Choose a small set of public pages with a clear reader task and record their current access rules.
2. Verify crawler identity against the operator’s current documentation. Preserve the timestamp, response, and URL scope.
3. Capture relevant answers under stated conditions and open each cited destination.
4. Inspect arrival and completion evidence separately. Record missing evidence explicitly rather than entering zero.
5. Review any proposed access change against its intended purpose, then annotate the deployment and observation window.

Use crawl ratios as context, not a substitute for outcomes

A crawl-to-referral ratio combines two events with different collection limits. A change in either side can move it. The crawl-to-referral guide explains those limits; the OpenAI publisher-controls guide separates search, training, and user-directed requests.

The decision to permit discovery should reflect the publication’s goals and access policy. Judge the result with evidence from the stage that matters to the business. A discovered page is a meaningful intermediate result; a traffic or revenue claim needs its own measurement.