Accessibility Tree vs HTML vs Screenshot: Which Page Representation Helps an AI Agent?
A preregistered protocol for comparing accessibility-tree, static HTML, rendered HTML, and screenshot representations on identical browser-agent tasks.
Protocol status: This page defines a test and contains no comparative finding. Cloudflare’s July 7, 2026 changelog confirms a Browser Run /accessibilityTree endpoint that returns roles, names, states, values, and hierarchy without screenshot pixels or full HTML. The existence of that representation does not establish that it is more accurate, cheaper, or faster for an AI agent.
The study must use permissioned pages and hold the model, task, page state, tools, output requirements, and stopping rule constant. A result belongs to the tested pages and tasks; it is not a universal ranking of accessibility trees, HTML, or vision.
Preregister pages and tasks
Select pages you control or are authorized to test: forms, menus, tables, modal dialogs, dynamic filters, custom widgets, charts, and pages with known accessibility defects. Define tasks as observable outcomes such as opening a menu, choosing an option, extracting a table value, submitting valid data, or stopping when permission is required.
Freeze viewport, browser version, cookies, authentication, locale, network fixtures, page version, initial focus, and dynamic data. Create hidden answer keys for target controls and expected final states before seeing model behavior.
| Representation | Preserve | Known blind spot to record |
|---|---|---|
| Accessibility tree | Roles, names, states, hierarchy | Visual-only meaning and canvas content |
| Static HTML | Source markup and attributes | Runtime state and rendered layout |
| Rendered HTML | Current DOM and properties | Pixels, occlusion, visual affordance |
| Screenshot | Rendered pixels | Hidden semantics and exact control state |
Build paired representation fixtures
Capture accessibility tree, raw static HTML, rendered DOM serialization, and screenshot from the same page state. Record raw bytes, encoded input size, model tokens where available, preprocessing, truncation, alt text, OCR, and any transformation that removes or invents information.
Randomize representation order inside pairs. Prevent earlier attempts from changing the page for later ones, and use a clean session or deterministic reset. Save tool calls, wrong actions, retries, timeouts, stops, and complete failures instead of retaining only successful trajectories.
Score actions and accessibility defects
Score task completion, number of actions, invalid actions, recovery, latency, input size, output size, and total cost separately. Review whether the agent selected the right control for the right reason, not merely whether the page eventually reached the target state.
When the accessibility tree lacks a name, role, state, or relationship needed for the task, record both the agent outcome and the underlying accessibility defect. Do not count a screenshot workaround as evidence that the semantic defect is harmless to users.
Report the sample without declaring a universal winner
Publish the pages or reproducible fixtures, task definitions, environment, representation builders, model, prompts, run count, stopping rule, raw outcomes, exclusions, and adjudication notes. Show absent and failed results alongside successes.
A valid conclusion might say that one representation performed differently on this bounded task set. It should not say that an accessibility tree replaces visual evidence or that screenshots replace semantic markup. Mixed representations may later deserve a separate, preregistered study.
Use the small experiment method for the observation card and the tool evaluation guide to keep one score from hiding tradeoffs.
Ask a question or join the discussion