Cloudflare /crawl Now Enforces Content Signals: Test reference vs full

What changed on August 31

Cloudflare’s August 31, 2026 changelog says /crawl now accepts a contentUse parameter with two values, ordered from less to more permissive: reference and full. The documented default is full. If the target’s Content Signals declaration permits less use than the request declares, Cloudflare rejects the crawl with a 400 response.

Cloudflare also documents crawlPurposes values such as search, ai-input, and ai-train. Purpose and maximum use answer different questions. A purpose describes why the request is being made; contentUse declares the maximum level of use. Preserve both in the test record.

Why the full default needs an explicit review

A default is convenient for an API client, but it can be too permissive for a team that intends only to retrieve references. If an integration sends no contentUse, the documented request level is still full. That can turn an otherwise reachable target into a 400 conflict when the publisher declares reference.

Set the field deliberately in code, configuration, and logs. Treat a missing value as a failed preflight if the system’s actual job is reference retrieval. This makes the declared use auditable and prevents a library upgrade or new code path from quietly inheriting the broader default.

{
  "url": "https://example.com",
  "crawlPurposes": ["search"],
  "contentUse": "reference",
  "formats": ["markdown"]
}

The payload above is an implementation pattern, not a result from SearchEngineAnswer. Use a target you control and the current Cloudflare endpoint documentation before running it.

Run a controlled four-state test

Create two fixture paths on a domain you control. Publish a full ceiling for one and a reference ceiling for the other. Send both reference and full requests to each path, one request at a time. Save the exact robots.txt response and timestamp with every API response because a later policy edit can otherwise make the evidence impossible to reproduce.

1. Confirm the fixture URL returns the intended public content without authentication.
2. Fetch and archive the current robots.txt plus response headers.
3. Send the request with an explicit crawlPurposes and contentUse.
4. Record status, error body, request ID where available, and UTC time.
5. Repeat only after changing one variable.
6. Classify the row as allowed, use-level conflict, access error, endpoint error, or inconclusive.

Download the Cloudflare crawl Content Signals test matrix (CSV). The file contains eight EXAMPLE-REMOVE rows. Replace them with controlled fixtures; do not publish tokens, account identifiers, private URLs, or response bodies containing sensitive data.

Assign the publisher and client controls separately

The publisher owns the declaration served from the target site. The client owner controls the purpose and use level sent to Cloudflare. A governance or legal reviewer may define the permitted use, but engineering still has to encode it and preserve evidence. Put those responsibilities in the runbook so a 400 conflict is not “fixed” by whichever team notices it first.

Classify a 400 before changing the publisher signal

Do not weaken a site’s Content Signal merely to make a client request succeed. First verify the target URL, account permissions, payload shape, and current documentation. Then compare the saved publisher ceiling with the request’s declared level.

What this control does not prove

Content Signals is a trust-based declaration. The Cloudflare check can stop a conflicting /crawl request, but it cannot establish the caller’s later behavior, legal rights, model-training use, deletion practice, or compliance by other services. A successful reference request also does not prove that a page will be indexed, cited, ranked, summarized, or shown to a user.

Keep access, declared use, downstream processing, answer appearance, citation, and referral as separate states. The publisher-controls outcome guide uses the same discipline for AI crawlers, and the retrieval research protocol shows why a visible source relationship should not be inflated into a hidden-system claim.

Source, method, and update note

Primary sources: Cloudflare’s August 31 changelog, the Browser Rendering crawl guide, the crawl API reference, and the Content Signals specification site, checked September 1, 2026.

Method: We converted the documented two-level contract into a four-state fixture matrix and separated expected outcomes from observations. SearchEngineAnswer did not run a Cloudflare account test for this article and reports no measured success or rejection rate.

Recheck trigger: Revisit this guide if Cloudflare changes the default, adds another use level, alters the error contract, or Content Signals publishes a material specification revision.