ChatGPT-User and robots.txt: What You Can—and Cannot—Block

Start with the three documented OpenAI agents

OpenAI says the OAI-SearchBot and GPTBot settings are independent. A publisher can allow potential search discovery while disallowing potential training use. OpenAI also notes that one crawl may serve multiple allowed uses to avoid duplicate fetching, so a request count cannot reveal the eventual downstream purpose.

Choose the policy outcome before writing directives

A useful policy starts with the content class and business decision, not a copied blocklist. Public reporting, documentation, subscriber previews, licensed archives, account pages and private tools do not share the same access promise.

• Public editorial pages: decide separately whether search discovery and potential training are acceptable.
• Paid or licensed content: keep the protected version behind authorization; a voluntary crawler directive does not enforce an entitlement.
• Account and private pages: require authentication and verify that caches, feeds and error pages do not expose protected data.
• Public tools and APIs: use quotas, input limits, abuse controls and clear terms because user-triggered fetching can create real load.

Use the broader AI crawler field guide when your policy covers several vendors. Use the allow-or-block decision guide when the unresolved issue is commercial rather than technical.

Use the smallest robots.txt rule that matches the decision

This example allows OpenAI’s documented search crawler while disallowing its documented training crawler:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

That file does not make a private URL safe, and it does not guarantee appearance in ChatGPT. It expresses a crawler preference. Test it against the exact public hostname and path you serve, including CDN behavior. If you block a crawler from fetching a page, do not expect it to discover a noindex directive placed only inside that blocked page.

Do not add a ChatGPT-User rule and assume the problem is solved. When the concern is expensive endpoints, licensed documents or account data, enforce the boundary where the content is served.

Verify the request before trusting the name

A user-agent string is easy to copy. Preserve the complete user agent, source IP, timestamp, hostname, path, method, response status, bytes, request rate and authentication context. Then compare the source with OpenAI’s current published IP ranges and account for reverse proxies that may hide the original address.

Our crawler identity verification workflow covers CIDR matching, failure handling and last-known-good data. When identity remains uncertain, prefer a narrow challenge, observation rule or rate limit over a permanent site-wide exception.

Run four tests, not one

1. Policy test: fetch /robots.txt publicly and confirm the intended group and path precedence.
2. Delivery test: request a representative public page and confirm a stable 200 response without a challenge page.
3. Protection test: request protected content without authorization and confirm the origin denies it even when the user agent changes.
4. Failure test: simulate unavailable identity data and excessive rate; the system should fall back to the ordinary public policy, not grant privilege.

OpenAI says its search systems may take about 24 hours to adjust after a robots change. Treat that as a product timing note, not a promise that a page will appear. Record the change time, edge tested, response and recheck time so a later observation is attributable.

Download the OpenAI crawler control decision ledger. Every sample row is marked EXAMPLE-REMOVE; replace it before using the file as a site record.

Know what the evidence cannot prove

A verified request and 200 response prove delivery to that requester at that moment. They do not prove that a later answer used the page, that a citation was earned, or that a visibility change came from the robots edit. Measure discovery, citations, explicit referrals and outcomes as separate layers. The ChatGPT SEO operating guide shows how to keep those observations separate.

Verification note: we rechecked the linked OpenAI documentation and IP-range publication on September 3, 2026. Vendor names, roles and network ranges can change; attach a retrieval date to every operational rule.

Primary documentation

• OpenAI: overview of OpenAI crawlers
• OpenAI: publishers and developers FAQ
• OpenAI: OAI-SearchBot published IP ranges
• OpenAI: ChatGPT-User published IP ranges