Search, Agent and Training Bots: A Practical Access Policy for Publishers
Build a crawler policy by purpose and named bot, with separate controls for search discovery, user-directed agents, training, verification, and server protection.
Published August 9, 2026: This policy framework uses current crawler documentation from Google, OpenAI, Anthropic, Cloudflare, and the Robots Exclusion Protocol.
A publisher should not answer “Do we allow AI bots?” with one site-wide yes or no. Search crawlers, user-triggered agents, and training crawlers perform different jobs. They can use different tokens, create different server loads, and lead to different outcomes.
The practical policy is to decide by purpose and named crawler: allow, rate-limit, challenge, or block. Document the business reason, the technical control, and the verification method for each decision.
Three jobs, not one bot class
| Purpose | Typical job | Decision question |
|---|---|---|
| Search | Discover or retrieve pages for search answers | Do we want this search product to find and present our pages? |
| Agent | Fetch a page because a user asked the agent | Can we serve user-directed retrieval safely and within capacity? |
| Training | Collect material for model development | Does our policy permit this use? |
OpenAI separates OAI-SearchBot from GPTBot. Anthropic separately documents ClaudeBot, Claude-User, and Claude-SearchBot. Cloudflare’s taxonomy similarly distinguishes search, AI assistants, and training. The names can change, so the policy should link to current owner documentation rather than rely on a copied list forever.
robots.txt is a request policy
RFC 9309 standardizes how crawlers retrieve and interpret robots.txt rules. It does not turn robots.txt into authentication or access authorization. A compliant crawler may honor the file; a hostile or misidentified client may not. Keep confidential material behind real access controls.
Use robots.txt for declared crawler policy, then use the CDN, web application firewall, rate limits, or server rules for enforcement and capacity protection. Do not block shared infrastructure or broad IP ranges until you understand which legitimate services could be affected.
Google recommends verifying Googlebot through reverse and forward DNS or its published IP ranges. Other operators publish their own verification methods. A user-agent string alone is not proof of identity.
Build the policy ledger
- Inventory named crawlers. Record the operator, token, purpose, documentation URL, and date checked.
- Choose the business outcome. Search discovery, user assistance, training permission, or server protection should be explicit.
- Set the control. Allow, disallow, rate-limit, challenge, or require authentication.
- Define verification. Use documented IP, DNS, signature, or platform verification where available.
- Observe logs. Compare declared policy with actual requests, response codes, crawl rate, and resource use.
- Assign an owner. Review quarterly and after material crawler-policy changes.
For every rule, write down the consequence you expect. Blocking a training crawler should not be described as blocking search unless the product owner documents that relationship. The OpenAI publisher-controls guide shows why these outcomes need separate rows.
A conservative default
Allow the search products you deliberately want, provided your pages are public and your infrastructure can serve them. Evaluate user-triggered agents separately because their requests may be irregular. Make a documented choice about training crawlers based on your rights, commercial policy, and risk tolerance.
Rate-limit abnormal behavior, protect login and private paths, and keep an incident path for sudden load. If the policy changes, annotate the date before interpreting visibility or referral changes. Access is the first layer of the AI visibility measurement crosswalk; it is not proof of an appearance.
Ask a question or join the discussion