Test AI Crawler Markdown and Accept Headers

The direct answer

Do not build and maintain a second Markdown version of your site merely because AI crawlers might prefer it. First inspect verified requests for an explicit Accept: text/markdown header. Cloudflare now documents HTML-to-Markdown content negotiation, but that proves a server capability—not that GPTBot, ClaudeBot, PerplexityBot, or other agents request Markdown, and not that Markdown improves citations.

Key Moments

Support and crawler use are different claims A server can offer Markdown even when a bot never requests it.
Log the full negotiation Capture Accept, Content-Type, Vary, cache status and a body hash.
Repeat requests from one client The second pass reveals whether cache variants remain isolated.
Control variant growth Header order and normalization can create more keys than expected.
Fresh Cloudflare evidence preserved the boundary Four live pages negotiated Markdown, while no crawler behavior was inferred.

The useful question is operational: which verified agents ask for which representation, what response do they receive, and does the change produce a measurable downstream effect? This protocol records that evidence without confusing a smaller response with a visibility gain.

What Cloudflare actually documents

Cloudflare’s Markdown for Agents documentation describes content negotiation: a client sends Accept: text/markdown, and an eligible Cloudflare-served page can return Markdown instead of HTML. The response can include token-count metadata, preserve JSON-LD, and use Vary: Accept so caches keep representations separate.

Capability is not adoption or SEO impact
Question	What is verified	What remains unproven
Can a server return Markdown?	Yes, when the documented Cloudflare feature is enabled and the request negotiates it.	Whether every origin, plan, cache, and page behaves identically.
Do major AI crawlers ask for it?	Not established by the product documentation.	Agent-by-agent request frequency in real logs.
Is it smaller?	Often, because presentation markup can be removed.	The reduction on your own templates and edge cases.
Does it improve citations?	No ranking or citation lift is documented.	Retrieval, citation, referral, or ranking impact.

A current r/TechSEO discussion is the lead for this investigation. It is useful practitioner context, not proof of crawler behavior.

Build a request ledger before changing the site

Record one row for every candidate crawler request. Keep raw logs privately, then publish only aggregated or redacted evidence. An IP address can be operationally sensitive; do not expose visitors or security controls in a public dataset.

Minimum request-level evidence
Field	Why it matters	Example state
Timestamp and URL class	Separates one burst from sustained behavior and prevents template mix-ups.	Article, archive, tool, image
Claimed user agent	Shows the identity string, but does not authenticate it.	GPTBot, OAI-SearchBot, ClaudeBot
Identity result	Prevents spoofed user agents from contaminating the test.	Verified, failed, unresolved
`Accept` header	Shows whether Markdown was explicitly requested.	`text/markdown`, wildcard, HTML
Returned content type	Confirms what the server actually served.	`text/html` or `text/markdown`
Status, bytes, cache	Finds failures and quantifies transport differences.	200, 42 KB, HIT
Readable-content parity	Catches links, tables, captions, or structured data lost in conversion.	Pass, partial, fail

Use the vendor’s official verification mechanism before labeling a request as genuine. A user-agent string is an assertion anyone can copy. Our companion guide, How to Verify an AI Crawler Before You Allow It, provides a vendor-by-vendor matrix.

A four-stage test that avoids false conclusions

1. Preserve the HTML baseline

Before enabling negotiation, save representative HTML responses for articles, tables, code blocks, author pages, tools, and archives. Record response bytes, main-text characters, links, image alternatives, headings, canonical URL, robots directives, and JSON-LD. The baseline is your rollback reference.

2. Enable negotiation on a bounded cohort

Start with a small set of stable URLs. Confirm that ordinary browsers still receive HTML, that the cache varies on Accept, and that unsupported or malformed requests fail safely. Do not put a sitewide rewrite into production before checking the pages most likely to break.

3. Verify representation parity

Compare HTML and Markdown for facts, citations, links, lists, tables, image context, author identity, publication dates, and structured data. A smaller file is a loss if the converted representation removes the evidence or navigation an agent needs.

4. Observe demand and downstream outcomes separately

Report the share of verified requests that explicitly ask for Markdown. Then, in a separate table, track AI citations, brand mentions, referral sessions, and crawl failures. Do not attribute a change to Markdown unless you have a contemporaneous control or a sufficiently stable baseline. Bot demand, transport efficiency, retrieval, citation, and referral are five different outcomes.

When enabling Markdown is defensible

Enable or continue: verified agents request it often enough to matter, parity tests pass, caching is correct, and operational cost stays low.
Keep as a limited experiment: response savings are meaningful but crawler demand or downstream value remains uncertain.
Stop or repair: important links, tables, authorship, structured data, or content disappear; caches cross representations; or errors increase.

This is not an invitation to create a second editorial system. One canonical source of truth should generate both representations. Maintaining independent HTML and Markdown pages introduces drift, duplicate URLs, and correction problems.

Sources and method

This protocol was prepared August 29, 2026 from Cloudflare’s product documentation and a current practitioner question. We treat Cloudflare as authoritative for its feature behavior and the community thread only as a discovery signal. We found no primary evidence that major AI crawlers universally request Markdown or that serving Markdown increases search rankings or AI citations. Those remain test questions.

Add cache correctness to the crawler test

A server can negotiate Markdown correctly at the origin and still deliver the wrong representation from cache. Cloudflare’s September Vary Cache Rules make this failure mode easier to control, but they also make the test more specific.

For each request variant, record five fields together: Accept, final Content-Type, Vary, cache status and a body hash. Send the HTML and Markdown requests twice from the same client. The second pass should show the expected cache behavior without crossing the bodies.

curl -sS -D markdown.headers -H 'Accept: text/markdown' https://example.com/page -o markdown.body
curl -sS -D html.headers -H 'Accept: text/html' https://example.com/page -o html.body
shasum -a 256 markdown.body html.body

Control variant growth before enabling it broadly

Accept values can include multiple formats in different orders. Missing headers and normalization rules can create more cache variants than the two formats a publisher intended. Cloudflare’s own example shows that combining language and content preferences can exceed the simple multiplication of visible choices.

Choose a custom cache key when one stable request property always determines the resource. Use Vary when the origin consistently declares which request fields select the response. Bypass caching when neither rule can guarantee correctness. The right decision is operational, not a crawler preference.

Our fresh observation separates support from use

On September 23, we made 25 requests across five Cloudflare URLs. Four live pages returned Markdown only for the explicit text/markdown request, and all four Markdown responses were cache hits with Vary: accept. One moved documentation URL returned 404 in every variant.

This proves that those live pages supported explicit Markdown negotiation at that moment. It still does not prove that a named AI crawler requests Markdown. To establish crawler behavior, inspect your server logs for its actual Accept header, then reproduce that request. Our expanded Cloudflare content-format observation contains the page-level results, while Cloudflare’s Vary documentation explains the cache controls.

Keep learning

Continue this topic

Control access

Do AI Crawlers Request Markdown? Test the Accept Header First