Web Bot Auth for Publishers: Verify Signed AI Agents Without Breaking Crawlers
Test signed AI-agent verification at the CDN, WAF, or origin while preserving IP, reverse-DNS, and user-agent fallback for unsigned requests.
Published August 9, 2026: Google is testing Web Bot Auth with a subset of requests from some AI agents. The protocol uses HTTP message signatures so a publisher or infrastructure provider can authenticate participating requests as a named agent such as https://agent.bot.goog.
This is an experimental identity signal, not a complete crawler policy. Google says not all user agents use Web Bot Auth and not every request from participating agents is signed. Publishers that test it must keep established IP, reverse-DNS, and user-agent verification as a fallback—and must still decide whether an authenticated agent is allowed to fetch the requested resource.
What Web Bot Auth proves—and what it does not
A valid signature can establish that a request was signed by the holder of a key associated with the asserted agent identity. That is stronger than trusting a user-agent string, which any client can copy. It can help a CDN, WAF, or origin distinguish participating automated traffic from an impersonator.
Authentication does not establish the publisher’s access decision. A correctly signed agent can still request a blocked path, exceed a rate limit, violate a contractual boundary, or encounter a resource that robots.txt disallows. Identity, purpose, permission, and outcome belong in separate fields.
The protocol itself remains a draft IETF effort and can change. A production design should hide verification behind a versioned component rather than spread draft-specific parsing across application code.
A publisher verification flow
- Receive the request without trusting its user-agent label.
- Read the
Signature-Agent,Signature, andSignature-Inputheaders for the relevant label. - Fetch the agent’s key directory from the documented
.well-knownendpoint. - Cache keys according to
Cache-Controland remove keys no longer present. - Verify the HTTP message signature according to RFC 9421 and its expiry rules.
- Map the authenticated identity to the publisher’s purpose and access policy.
- Fall back to established IP, reverse-DNS, and user-agent verification when a request is unsigned.
- Log the evidence and resulting allow, limit, challenge, or deny decision.
Google’s current example uses keys at https://agent.bot.goog/.well-known/http-message-signatures-directory and an asserted agent value of g="https://agent.bot.goog". Treat those as current implementation facts, not a universal pattern for every automated client.
Design the fallback before enabling enforcement
| Observed request | Identity result | Next action |
|---|---|---|
| Valid signature | Authenticated participating agent | Apply that agent’s access and rate policy |
| Unsigned participating UA | Unknown from WBA alone | Use IP, rDNS, and UA fallback |
| Invalid signature | Untrusted or broken request | Deny or challenge; preserve evidence |
| Valid identity, blocked path | Known caller | Enforce publisher policy |
| Key directory unavailable | Verification temporarily incomplete | Use cached keys and defined failure mode |
Do not create a fail-open path by accident. Define what happens when the key directory times out, a clock is skewed, a signature expires, a cached key disappears, or an intermediary changes a covered header. Test those states in staging.
A cautious rollout checklist
Start in observe-only mode. Record signed, unsigned, valid, invalid, and unverifiable requests by path and agent. Compare the results with the existing search, agent, and training-bot policy. Only then place authenticated traffic into a separate policy lane.
Ask the CDN or WAF whether verification is native, which identities it recognizes, how key rotation is handled, whether raw signature evidence is logged, and what happens during an upstream outage. Keep a small direct verifier for test fixtures even if the provider handles production traffic.
Review the protocol and Google documentation on a fixed schedule because the specification is experimental. The durable architecture is a normalized identity record with version, method, evidence, confidence, policy, and decision—not a one-off allowlist toggle.
Ask a question or join the discussion