Cloudflare’s September 15 AI Crawler Defaults: What Publishers Must Check

What Cloudflare announced

The change applies to new domains onboarding to Cloudflare, not automatically to every existing zone. Cloudflare separates three AI-related behaviors: Search, Agent, and Training. Its announced ad-page defaults allow Search and block Agent and Training.

That classification is behavior-based, not a promise that one user-agent always has one purpose. Cloudflare says a bot can be multipurpose and that the most restrictive applicable setting will decide whether it is allowed. The result can be broader than a publisher expects from a label such as “block training.”

Audit before September 15

1. Export the current configuration. Record the zone, active bot products, legacy Block AI Bots state, custom WAF rules, exceptions, and owner.
2. Map ad-bearing templates. Include article, category, search, tool, and archive pages; do not assume the homepage represents the site.
3. List essential crawlers. Start with Googlebot and Bingbot, then add services required for previews, monitoring, feeds, or commerce.
4. Verify identity. A user-agent string can be spoofed. Use the operator’s documented IP or reverse-DNS method and Cloudflare’s verified-bot signals where appropriate.
5. Test the final request path. Check status, challenge page, response headers, robots controls, canonical, and visible main content from representative URLs.
6. Set an alert. Watch verified-bot 403s, challenge responses, crawl-rate changes, and Search Console or Bing crawl anomalies after the release date.

The existing Cloudflare Googlebot 403 incident guide shows why an intended AI control must be checked at the delivered response. Use the broader Search, Agent, and Training bot policy to record the decision rather than leaving it inside a dashboard toggle.

Run a small response matrix

Choose at least one ad page and one non-ad page from every important template. Record the same fields before and after the change: URL, crawler identity, behavior classification, applied rule, HTTP status, challenge state, robots directive, canonical, body length, and whether the main content is present.

A passing browser visit is not a passing crawler test. A human session may carry cookies, JavaScript, or challenge clearance that the crawler does not have. Compare raw responses and Cloudflare event logs, then reproduce the result without the authenticated browser state.

If a required search crawler is blocked, narrow the rule or add a verified exception whose scope and owner are documented. Do not whitelist an arbitrary user-agent string, and do not remove all bot controls merely to repair one classification conflict.

What the announcement does not prove

• Blocking a Training-classified request does not prove that the operator would have used the response for model training.
• Allowing a Search-classified crawler does not guarantee discovery, indexing, ranking, citation, or referral traffic.
• A Cloudflare verified-bot classification does not replace the bot operator’s own documentation about intended use.
• The September defaults do not remove the publisher’s responsibility to review existing and custom rules.

Treat this as a release change with a rollback path. Preserve the old settings, name the person who can reverse the change, and run the technical SEO launch checklist on any production adjustment.

Primary documentation

• Cloudflare: Your site, your rules—new AI traffic options for all customers
• Cloudflare verified bots and behavior taxonomy