Do AI Crawlers Request Markdown? Test the Accept Header First

The direct answer

Do not build and maintain a second Markdown version of your site merely because AI crawlers might prefer it. First inspect verified requests for an explicit Accept: text/markdown header. Cloudflare now documents HTML-to-Markdown content negotiation, but that proves a server capability—not that GPTBot, ClaudeBot, PerplexityBot, or other agents request Markdown, and not that Markdown improves citations.

The useful question is operational: which verified agents ask for which representation, what response do they receive, and does the change produce a measurable downstream effect? This protocol records that evidence without confusing a smaller response with a visibility gain.

What Cloudflare actually documents

Cloudflare’s Markdown for Agents documentation describes content negotiation: a client sends Accept: text/markdown, and an eligible Cloudflare-served page can return Markdown instead of HTML. The response can include token-count metadata, preserve JSON-LD, and use Vary: Accept so caches keep representations separate.

A current r/TechSEO discussion is the lead for this investigation. It is useful practitioner context, not proof of crawler behavior.

Build a request ledger before changing the site

Record one row for every candidate crawler request. Keep raw logs privately, then publish only aggregated or redacted evidence. An IP address can be operationally sensitive; do not expose visitors or security controls in a public dataset.

Use the vendor’s official verification mechanism before labeling a request as genuine. A user-agent string is an assertion anyone can copy. Our companion guide, How to Verify an AI Crawler Before You Allow It, provides a vendor-by-vendor matrix.

A four-stage test that avoids false conclusions

1. Preserve the HTML baseline

Before enabling negotiation, save representative HTML responses for articles, tables, code blocks, author pages, tools, and archives. Record response bytes, main-text characters, links, image alternatives, headings, canonical URL, robots directives, and JSON-LD. The baseline is your rollback reference.

2. Enable negotiation on a bounded cohort

Start with a small set of stable URLs. Confirm that ordinary browsers still receive HTML, that the cache varies on Accept, and that unsupported or malformed requests fail safely. Do not put a sitewide rewrite into production before checking the pages most likely to break.

3. Verify representation parity

Compare HTML and Markdown for facts, citations, links, lists, tables, image context, author identity, publication dates, and structured data. A smaller file is a loss if the converted representation removes the evidence or navigation an agent needs.

4. Observe demand and downstream outcomes separately

Report the share of verified requests that explicitly ask for Markdown. Then, in a separate table, track AI citations, brand mentions, referral sessions, and crawl failures. Do not attribute a change to Markdown unless you have a contemporaneous control or a sufficiently stable baseline. Bot demand, transport efficiency, retrieval, citation, and referral are five different outcomes.

When enabling Markdown is defensible

• Enable or continue: verified agents request it often enough to matter, parity tests pass, caching is correct, and operational cost stays low.
• Keep as a limited experiment: response savings are meaningful but crawler demand or downstream value remains uncertain.
• Stop or repair: important links, tables, authorship, structured data, or content disappear; caches cross representations; or errors increase.

This is not an invitation to create a second editorial system. One canonical source of truth should generate both representations. Maintaining independent HTML and Markdown pages introduces drift, duplicate URLs, and correction problems.

Sources and method

This protocol was prepared August 29, 2026 from Cloudflare’s product documentation and a current practitioner question. We treat Cloudflare as authoritative for its feature behavior and the community thread only as a discovery signal. We found no primary evidence that major AI crawlers universally request Markdown or that serving Markdown increases search rankings or AI citations. Those remain test questions.