A URL Can Return HTTP 200 and Still Fail Before the Client Reads HTML

Method and environment

The fixture used Node’s HTTP server on loopback. Each path returned 200 OK, a constant HTML document containing “HTML reached,” a normal content-type field, and one synthetic x-sea-fixture response field. Only that field’s value length changed: 8,192; 15,000; 16,000; 16,500; 17,000; and 32,768 bytes.

Two clients fetched every path: curl 8.7.1 using its normal settings and Node.js v26.0.0 using http.get(). Node was also retested at 17,000 and 32,768 bytes with maxHeaderSize set to 65,536. Success required both a response object and the HTML marker in the body. The fixture, runner, version output, and raw matrix are retained with this article package.

This is a local implementation test, not a measurement of Googlebot, Bingbot, CDN behavior, or every build of curl or Node. Header names, other fields, HTTP version, proxies, and software updates can move the exact failure point.

Results

Node documents a 16 KiB default maximum for HTTP headers in the current line. The failure began above that total message-header budget, not at an exact synthetic-field value guaranteed for every response. The field was 16,500 bytes, while the status line and other headers added overhead.

Why status 200 is not enough

HTTP semantics define what 200 means, but RFC 9110 does not impose one universal header-section size that every recipient must accept. Implementations set practical limits for resource use and security. CDNs and proxies set different limits again; Cloudflare, for example, documents a 128 KB request and response header limit after an October 2025 increase.

A monitoring service may report 200 because its client accepted the response. A different renderer, crawler, SDK, or serverless function can reject the same bytes. If that happens before content parsing, canonical tags, robots directives, links, structured data, and the main answer never reach the client—even though the origin log says 200.

A diagnostic runbook

1. Save the complete response. Record status, protocol, raw header bytes, individual field sizes, body bytes, timestamp, and network path.
2. Test more than one client. Use the production renderer or SDK as well as curl; preserve version and configuration.
3. Reduce one field at a time. Large cookies, CSP, Link, Set-Cookie, tracing, and cache metadata are common places to inspect.
4. Check every hop. Compare origin, CDN, WAF, proxy, load balancer, and client limits.
5. Confirm HTML was parsed. A status code is not evidence that the title, canonical, or body was read.
6. Fix the cause before raising limits. Increasing a client limit can validate the diagnosis, but unnecessary headers should still be removed.

Add a header-budget check to the technical launch checklist and preserve the release as a bounded observation using the small SEO experiment method.

What this says about crawlers

Google documents a separate two-megabyte processing limit per fetched resource that includes the response headers and body. That figure is not a published maximum header-section size and should not be converted into one. The local Node result also does not prove Googlebot shares Node’s parser or default.

The evidence supports a troubleshooting principle: reproduce the failing client, inspect the bytes before HTML, and avoid treating one successful fetch as universal accessibility. For search crawlers, pair server logs with the platform’s inspection tools and verified client identity.

Primary documentation

• RFC 9110: field limits
• Node.js HTTP documentation: maximum header size
• Cloudflare: HTTP header limit increase to 128 KB
• Google Search Central: bytes Googlebot processes