AI crawlers explained: Which bots index, train on, or fetch your website

What AI crawlers actually do

The phrase AI bot hides four different jobs. Treating them as one class leads to broken search discovery, ineffective training opt-outs, and firewall rules that trust a name anybody can copy.

Some operators use a single crawler for more than one purpose. Apple says Applebot supports search experiences and that crawled material may also be used to train Apple foundation models. Its separate Applebot-Extended token lets a publisher express a training-use preference while keeping Applebot available for search. Google documents a similar separation: Googlebot crawls for Search, while Google-Extended controls specified Gemini training and grounding uses without changing Google Search inclusion or ranking.

Major AI bots and controls compared

This registry covers major identities with current operator documentation or a verified directory record. It deliberately excludes copied names whose operator, purpose, or verification method could not be established. Recheck the linked documentation before deploying a long-lived rule.

Cloudflare also lists other AI-related identities, including DuckAssistBot, MistralAI-User and Google-CloudVertexBot. The correct response is not to paste every observed name into robots.txt. Add an identity only after you can document its operator, job, desired business outcome, verification method, and review owner.

Understand the platform families before optimizing

Google Search and Gemini controls

Googlebot is the ordinary crawler behind Google Search. Google says pages appearing in AI Overviews or AI Mode need no special AI schema or file: the supporting page must meet normal Search requirements and be eligible for a snippet. Blocking Googlebot to stop one Gemini use would also affect Search crawling.

Google-Extended solves a narrower policy problem. It is a robots.txt token controlling whether Google-crawled content may be used for future Gemini training and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It is not a user agent, so a log report claiming “Google-Extended visits” is mislabelled.

Google also documents user-triggered clients such as Google-Agent and Google-GeminiNotebook. These operate because a user requested an action or supplied a source and generally ignore robots.txt. Authentication, authorization and non-public URLs—not a robots file—must protect sensitive content.

OpenAI search and training controls

OpenAI separates OAI-SearchBot from GPTBot. OpenAI says OAI-SearchBot access is needed for inclusion in ChatGPT summaries and snippets; GPTBot controls potential training use. A publisher can therefore allow search discovery and disallow potential training. The detailed OpenAI publisher-controls guide separates crawl access, citations and referrals.

OpenAI also notes that a title and link can surface through third-party search or other discovery even when direct crawling is disallowed. If a page must not appear, a supported noindex instruction may be relevant—but the crawler must be able to read it. Blocking the URL in robots.txt while expecting the crawler to see a page-level noindex is contradictory.

Claude and Perplexity live fetchers

Anthropic documents three explicit roles: ClaudeBot for content that could contribute to model development, Claude-SearchBot for web search, and Claude-User for a fetch made at a user’s request. Anthropic says all three honor robots.txt and publishes source-IP information. It also warns that a network block can stop the crawler from reading a changed robots policy, so IP blocking is not a durable preference mechanism.

Perplexity makes a different commitment. PerplexityBot builds its search index and respects robots.txt. Perplexity-User retrieves a page for a user and generally ignores robots.txt. Perplexity publishes JSON IP ranges for both. This is why “block all AI bots in robots.txt” is not a security plan.

Bing, Apple, Amazon and Common Crawl

Bingbot is Microsoft’s standard crawler for the Bing index. That index can support search and AI experiences, but an individual Bingbot request does not prove a Copilot citation. Bing provides a public verification tool and crawl controls; use them instead of trusting the user-agent string.

Apple and Amazon both document multipurpose families. Applebot supports Apple search experiences, while Applebot-Extended expresses a foundation-model use preference. Amazon separates Amazonbot, Amzn-SearchBot and Amzn-User so publishers can distinguish model-development collection, search discovery and user-requested fetching.

CCBot is different again: Common Crawl creates an openly available web corpus. A fetch can lead to corpus inclusion that many downstream users may access, but the request cannot identify a particular model, training run, answer, or citation.

Choose an AI-bot access policy by purpose

Start with the business decision, not the syntax. For every identity, record whether access helps readers find public work, supports a product you value, conflicts with licensing or privacy obligations, creates unacceptable load, or remains unknown. The Search, Agent and Training bot policy provides the broader governance framework.

A sensible public publisher policy often allows search discovery, evaluates training separately, and keeps private or licensed material behind real access control. A documentation site that wants broad model use may choose differently from a subscription publisher. Neither choice can be copied safely without the content and business boundary.

Use robots.txt without treating it as a firewall

RFC 9309 standardizes how compliant crawlers find and interpret /robots.txt. The file expresses crawl preferences. It does not encrypt a page, revoke previously collected copies, bind unknown scrapers, or replace authentication.

The following example represents one possible policy: allow documented search crawlers while declining selected training or open-corpus uses. It is a template to review, not a universal recommendation.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

This file does not need explicit allow groups for Googlebot, Bingbot, Applebot or Amzn-SearchBot when a site’s existing wildcard group already permits the public paths. Add only rules that make the final policy easier to audit, and test how specific groups interact with existing wildcard groups.

For a mixed public and restricted site, exclude non-public path patterns from every relevant compliant crawler, but enforce the boundary with login and authorization:

User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /internal-search/
Disallow: /private-downloads/

Sitemap: https://www.example.com/sitemap.xml

Do not list secret paths merely to hide them; robots.txt is public. Do not block a URL and rely on a meta robots tag at that URL, because the blocked crawler cannot read it. Do not assume a training-bot rule covers a same-operator user fetcher or control token.

Verify real bots at the CDN, WAF and origin

A user-agent is a string supplied by the requester. A scraper can call itself Googlebot, GPTBot, or anything else. Discover traffic with the user agent, then authenticate or corroborate it with the strongest operator-supported method available.

1. Match the exact documented token. Preserve the full raw user agent and request time instead of normalizing every visitor into “AI bot.”
2. Check the network identity. Use official IP JSON, reverse-DNS plus forward verification, or the operator’s verification tool. Cache verified results rather than performing DNS work on every request.
3. Prefer signed identity when available. Web Bot Auth uses HTTP message signatures to prove the requester controls a registered key. The publisher verification guide explains how to stage it without breaking unsigned search crawlers.
4. Inspect rule precedence. A correct robots file cannot repair a CDN rule that challenges the bot before it reaches the origin. Check managed bot settings, custom WAF rules, rate limits, country blocks and cache variants.
5. Test the delivered response. Record status, redirects, content type, bytes, canonical, robots meta, body hash and whether the main content is present. A browser test with cookies is not equivalent to a clean crawler request.

Cloudflare’s verified-bot program can identify requests through signed identity, published IP ranges with stable user agents, or reverse DNS. “Verified” means the identity and behavior met Cloudflare’s criteria; it does not mean every publisher should allow every verified bot.

Optimize the website that permitted bots receive

There is no shared “AI index” switch. The durable technical work is ordinary web publishing performed carefully for the exact clients you permit.

1. Return the intended final status. Public canonical pages should normally resolve to a stable 200. Repair redirect loops, soft 404s, intermittent 5xx responses and bot-only 403 challenges.
2. Keep robots, meta robots and access policy consistent. Decide whether the page should be crawled, indexed, previewed and used for each documented purpose. Test the final public files after CDN transforms.
3. Expose the main answer in rendered HTML. Do not require a click, login, client-only state, canvas or fragile script merely to reveal the core text. Server-rendered or reliably rendered HTML lowers the failure surface.
4. Use stable canonical URLs. Point duplicate and parameter variants to the correct public source, link internally to that source, and keep XML sitemaps free of redirects, errors and non-canonical URLs.
5. Make important links crawlable. Use real anchor elements with descriptive text. Keep primary navigation, related guides and evidence sources reachable without a scripted search box.
6. Keep page identity visible. Show one clear title, author, publication or update date, section headings, organization context and a correction path. Accurate structured data can repeat that visible identity; it must not invent it.
7. Serve accessible media. Provide useful alternative text, captions or transcripts, correct dimensions and a stable media URL. Do not hide the only explanation inside an image.
8. Control page weight and crawl traps. Avoid infinite parameter spaces, faceted loops, calendar expansion and duplicate internal-search results. Monitor response size and timeouts for both browsers and verified bots.

Google explicitly says its AI features require no special schema or AI-specific file. The safer rule is broader: use documented schema that matches visible content, and do not expect llms.txt, a bot allowlist, or a new meta tag to override the target platform’s actual eligibility contract.

Optimize the content for safe answer use

Crawlability creates an opportunity to inspect a page. It does not make the page worth using. The editorial layer should help a reader—and any system quoting the passage—retain the claim’s subject, scope and evidence.

• Lead with the direct answer. Put the conclusion near the heading that names the question.
• Define the boundary. State the product, geography, population, date or scenario covered by the claim.
• Keep proof beside the claim. Prefer current operator documentation, standards, original research and reproducible first-hand evidence.
• Add a useful distinction. Explain what readers commonly confuse—for example, crawling versus indexing, or citation versus recommendation.
• Show the limit. Say what the evidence cannot establish and what would change the recommendation.
• Contribute information gain. Add a tested workflow, decision table, calculation, template, comparison or failure state instead of rearranging other summaries.

The citation-ready passage test turns those principles into a repeatable edit. The broader Answer Engine Optimization guide connects eligibility, clarity, support and measurement across answer surfaces.

Measure access, visibility and value separately

A useful dashboard preserves the chain instead of collapsing it into one AI-visibility score.

For logs, retain the verified operator, exact identity, URL, method, timestamp, status, bytes, cache state, country or network evidence, applied rule and body signature where policy permits. Group public article requests separately from feeds, assets, parameter traps and error pages. A high crawl count can reflect waste or repeated refetching rather than useful visibility.

Run a 45-minute AI crawler audit

1. Write the policy. List the bot identities, declared jobs, desired outcomes and content scope. Mark uncertain entries for research instead of guessing.
2. Fetch the public controls. Save robots.txt, representative meta robots tags, X-Robots-Tag headers, canonical tags and sitemap responses from the live site.
3. Compare edge and origin rules. Review CDN bot settings, WAF custom rules, rate limits, country blocks and hosting security plugins.
4. Inspect 30 days of logs. Find the documented tokens, then verify a sample with IP, DNS, signature or operator tooling. Preserve spoofed and failed requests separately.
5. Test representative URLs. Include the homepage, an article, a category, a tool, a media URL, a redirect, a 404 and a restricted page. Record the final response rather than the intended configuration.
6. Review answer passages. Copy the strongest paragraph into a blank document and check whether its subject, boundary, evidence and limit still make sense.
7. Set measurement and maintenance. Assign an owner, next review date, documentation-change trigger and alert for unexpected verified-bot 403s or server-load spikes.

Stop the audit when the team cannot explain why an identity is allowed, blocked or rate-limited. An unowned bot list becomes stale infrastructure surprisingly quickly.

Avoid these common mistakes

• Calling every bot an indexer. Training collectors and user fetchers may never build a public search index.
• Blocking the wrong identity. Blocking GPTBot does not block OAI-SearchBot; blocking ClaudeBot does not automatically express the same choice for Claude-SearchBot.
• Counting control-token visits. Google-Extended and Applebot-Extended do not send HTTP requests.
• Trusting the user-agent alone. A spoofed bot can bypass a naive allow rule or poison an analytics report.
• Using robots.txt for secrets. Protect private, paid and personal content with authorization at the application or origin.
• Allowing robots but blocking at the edge. Challenge pages, 403s and rate limits can make the delivered result contradict the policy file.
• Promising AI rankings after a crawl fix. Access is one technical prerequisite, not proof of retrieval, citation or recommendation.
• Copying a permanent allow/block list. Bot identities, products, purposes and controls change; every rule needs an owner and review trigger.

Primary documentation

• RFC 9309: Robots Exclusion Protocol
• Google: Common crawlers and Google-Extended
• Google: User-triggered fetchers
• OpenAI: Publishers and developers FAQ
• Anthropic: Web crawler controls
• Perplexity: Crawler documentation and IP ranges
• Bing: Crawler registry
• Bing: Verify Bingbot
• Apple: About Applebot and Applebot-Extended
• Amazon: Amazonbot, Amzn-SearchBot and Amzn-User
• Common Crawl: CCBot
• Cloudflare: AI bot reference
• Cloudflare: Verified bots

The evidence ends at documented access, observed responses and platform reports. No crawler configuration can guarantee that an answer engine will retrieve, cite, recommend or send traffic to a page.