Cloudflare Will Set Different AI Bot Defaults for Search, Agent and Training

What Cloudflare says will change

Cloudflare’s documentation separates automated traffic into Search, Agent, and Training purposes. Each purpose can be configured as Allow, Block pages with ads, or Block all. The controls are available under Security Settings in the AI bot policies configuration.

For domains added on or after September 15, Cloudflare documents these defaults: Search allowed; Agent blocked on pages with ads; Training blocked on pages with ads. Customers can opt out before the date. The legacy “Block AI Bots” control is scheduled for deprecation on September 15 as the purpose-based settings take over.

Existing domains need an explicit read

The announcement describes defaults for new domains. It does not establish that an existing domain’s current policy will be rewritten to match. Open the domain’s AI bot policy screen and record all three settings rather than copying the new-domain table into an audit.

Also capture whether the domain used the legacy Block AI Bots switch, when it was added to Cloudflare, and whether a person or API last changed the policy. A screenshot is useful, but export or transcribe the values into a dated ledger so they can be compared after the rollout.

For content-use declarations at the crawl endpoint, see our Cloudflare /crawl Content Signals guide. That endpoint contract and the bot-management defaults are related publisher controls, but they are not the same enforcement surface.

Test mixed-purpose bots as their own case

Cloudflare says bots classified as both Search and Training are blocked under configurations that block AI training, including the legacy Block AI Bots control. This mixed-purpose rule is important because “Search is allowed” does not necessarily override a Training restriction.

Do not generalize the classification from a vendor name. Record the specific bot, Cloudflare’s observed category, request path, page ad state, rule matched, and action. If the purpose is unknown or classification changes, mark the row inconclusive until evidence is available.

The broader AI crawler decision guide helps align access choices with business goals, while the ChatGPT robots controls guide covers a product-specific layer outside Cloudflare’s classification.

Run the pre- and post-rollout audit

Download the Cloudflare AI bot pre/post audit (CSV). Its rows are marked EXAMPLE-REMOVE. The September 16 rows are test fixtures, not observations.

1. Before September 15, save the domain creation date and each purpose setting.
2. Record whether representative pages contain ads and how ad presence is detected.
3. Preserve the legacy-control state and any custom WAF or bot rules that can affect the same request.
4. After rollout, create a controlled new domain or zone only if your account and policy allow it.
5. Observe Search, Agent, Training, mixed-purpose, and unknown classifications separately.
6. Save request time, bot identity, classification, matched rule, action, and response evidence.
7. Compare expected defaults with observed configuration before testing network behavior.

A failed request does not by itself prove the AI bot policy caused the failure. DNS, origin availability, authentication, robots rules, rate limits, WAF rules, and custom firewall logic can produce similar symptoms.

Choose policy by purpose, not by fear

Search retrieval may create discovery or citation opportunities. Agent access may serve a user who asks a tool to fetch your page. Training may create a different value exchange. A single “AI bot” toggle hides those distinctions.

Assign an owner and written intention to each purpose. If ad-funded pages require a different policy, define how the site labels those pages and test the label. If training is never authorized, use the explicit block-all control instead of relying on the ads condition.

Do not promise that allowing Search will cause indexing, citation, ranking, or referral traffic. Access is only the first state in the chain.

What can be published before and after September 15

Before rollout, publish the documented defaults, your current configuration, and the planned test. Label future rows “expected” or “not observed.” After rollout, add an observed result only when the domain cohort, configuration, request evidence, and confounding rules are preserved.

If observed behavior differs from the documentation, report the mismatch and conditions rather than declaring the documentation wrong. Account rollout timing, domain age, configuration inheritance, or classification can explain a difference.

Source, method, and update note

Primary source: Cloudflare’s Block AI bots documentation, checked September 3, 2026.

Method: SearchEngineAnswer converted the documented future defaults into a pre/post audit. We have not observed the September 15 rollout and report no live block rate.

Recheck trigger: Verify after September 15, then update only with reproducible observations. Recheck sooner if Cloudflare changes the date, defaults, purpose definitions, mixed-purpose handling, or legacy-control deprecation.