ChatGPT-User and robots.txt: What Publishers Can Actually Control
Separate ChatGPT user-triggered visits from OpenAI search crawling and training crawling, then apply the right robots, firewall, and access controls to each job.
Direct answer: a publisher should not treat every OpenAI request as the same crawler. OpenAI documents OAI-SearchBot for ChatGPT Search discovery, GPTBot for crawling that may support model training, and ChatGPT-User for certain actions initiated by a ChatGPT or Custom GPT user. The first two have explicit robots.txt roles. OpenAI says robots.txt rules may not apply to ChatGPT-User because the request is user initiated.
That distinction changes the control you use. A robots rule can express a search or training preference. It is not a substitute for authentication, authorization, rate limits, or a firewall rule when the content itself must not be publicly retrieved.
Map the three OpenAI agents before writing rules
| Agent | Documented job | Primary publisher control | Do not assume |
|---|---|---|---|
| OAI-SearchBot | Surface sites in ChatGPT search features | robots.txt plus verification of published IP ranges | Allowing it guarantees appearance or a citation |
| GPTBot | Crawl content that may be used for foundation-model training | robots.txt training preference | Its rule also controls ChatGPT Search |
| ChatGPT-User | Some user-triggered page visits and external actions | Application access controls, firewall policy and rate limits | A robots.txt denial will always prevent the request |
OpenAI says the OAI-SearchBot and GPTBot settings are independent. A publisher can allow search discovery while disallowing training use. OpenAI also notes that one crawl may serve both allowed uses to avoid duplicate fetching, so raw request volume alone does not reveal the downstream purpose.
Write a policy by content class
Start with the page’s access promise, not the bot name. Public articles, documentation, subscriber previews, licensed archives, account pages and internal tools should not inherit one blanket rule.
- Public editorial pages: decide separately whether you want ChatGPT Search eligibility and whether you permit potential training use.
- Paid or licensed content: keep the protected version behind real authorization. A voluntary crawler directive does not enforce a commercial entitlement.
- Account and private pages: require authentication, deny unauthenticated requests, avoid exposing secrets in HTML, feeds or predictable files, and audit cache behavior.
- Public tools and APIs: use quotas, input limits, abuse controls and documented terms. A user-triggered agent can generate real application load.
The broader AI crawler guide maps other search, training and user agents. The founder decision guide helps turn the policy into a reversible business decision.
Verify a request before blocking it
A user-agent string is easy to copy. Before allowing or denying traffic based on identity, record the complete user agent, source IP, timestamp, host, path, method, status, bytes, rate and any authentication context. Compare the address with OpenAI’s current published ranges and review reverse-proxy or CDN transformations that may hide the original source.
| Evidence | Question | Failure signal |
|---|---|---|
| User agent | Does the full string match the documented pattern? | Only a copied product name appears |
| Network | Does the source match the current published range? | Address is unrelated or obscured |
| Behavior | Is the rate and path consistent with the claimed job? | Credential probing or destructive requests |
| Response | Did the request receive public, protected or error content? | Private data escaped through a cache or fallback |
When identity remains uncertain, use a narrow challenge, rate limit or observation rule rather than a permanent site-wide block. Preserve false positives and false negatives so the policy can be corrected.
Test the result at the right layer
After changing robots.txt, verify that the file returns HTTP 200 publicly, contains the intended directives, and is not altered between IPv4, IPv6, regions or CDN edges. OpenAI says search systems may take about 24 hours to adjust to a robots.txt change. That is a product statement, not a promise that your page will then be included.
For a firewall or authorization change, test the exact public, logged-in and agent paths. Confirm that protected content remains protected while ordinary readers and wanted crawlers still receive the correct status and page. Monitor 401, 403, 429 and 5xx rates separately.
Limit: server logs can show a request and response. They cannot prove how an answer system later used the content, whether the page was cited, or whether a policy change caused a visibility change.
Ask a question or join the discussion