How to Verify GPTBot and Other AI Crawlers

A bot name in a log is not proof of identity

Any client can send a user-agent string that says GPTBot, ClaudeBot, PerplexityBot, or Googlebot. Before changing a firewall rule, rate limit, robots policy, analytics segment, or paid-content boundary, verify the source using the operator’s official IP list or documented DNS procedure. Store three separate states: claimed, verified, and served.

This guide expands our earlier web-bot authentication framework with a current operational matrix for OpenAI, Anthropic, Perplexity, and Google.

Use the operator’s method, not one universal shortcut

A current r/TechSEO report raised the spoofing risk. The risk is technically plausible because user-agent strings are client-controlled, but the thread does not establish how often spoofing occurs across the web.

Keep a three-state crawler record

The separation catches two common errors. A genuine crawler can be blocked accidentally, while a spoofed crawler can receive privileged treatment because a rule trusted the name alone.

A safer verification workflow

1. Capture without granting: log timestamp, claimed agent, source IP, requested URL class, method, status, bytes, and response time.
2. Fetch official data on a schedule: download the vendor’s IP ranges over HTTPS, validate the response, retain the last known-good copy, and record its retrieval time.
3. Match addresses correctly: test IPv4 and IPv6 against CIDR ranges. Do not use string prefixes such as “starts with 20.”
4. Use DNS only where documented: reverse lookup alone is insufficient. Google’s process requires reverse DNS followed by a forward lookup that returns the original IP.
5. Fail closed for privileges: an unavailable vendor endpoint should not convert an unverified request into an allowed privileged request. Continue with the last valid list within a declared freshness window or use the default public policy.
6. Separate purpose policies: search discovery, training, user-initiated retrieval, and ads may use different agents. Express those decisions explicitly in robots, WAF, and application rules.

Cache verification results briefly to reduce repeated network work, but do not make the cache permanent. Vendors can change ranges. Monitor misses from previously verified agents before replacing an old list.

Copy this verification checklist

• Identify the exact agent and its documented role.
• Save the official verification URL and retrieval timestamp.
• Validate JSON or DNS results before updating rules.
• Support both IPv4 and IPv6 CIDR matching.
• Record claimed, verified, and served as separate fields.
• Test allow, block, rate-limit, and endpoint-failure cases.
• Keep a last-known-good list and rollback procedure.
• Review rules when a vendor adds or renames an agent.

Download or print the checklist from the article controls, then attach the completed record to the relevant WAF or robots change. A security decision should be reproducible by someone other than the person who wrote the rule.

Sources and evidence limits

We checked the linked operator documentation on August 29, 2026. Vendor documentation is authoritative for agent names, purposes, and published identity mechanisms; it cannot guarantee that every request bearing those names is genuine. The community posts that prompted this guide are anecdotal discovery signals and are not used to estimate spoofing prevalence.