Q2D-Web Shows Why AI Citations Depend on First-Stage Retrieval

The study in one research card

The authors are researchers at Perplexity AI. That matters because the benchmark reflects a production retrieval environment, but it also means readers cannot independently download the complete dataset and reproduce every reported result. The paper and leaderboard are public; the underlying evaluation collection is not.

Most of the queries are support queries

Q2D-Web does not evaluate only the neat question a person types. The agent turns the user’s message, conversation history and earlier search results into a primary query plus optional support queries.

This changes how an SEO or AEO practitioner should interpret the benchmark. A page may be relevant to an intermediate lookup even when it does not match the original wording. Entity disambiguation, background definitions, comparisons and adjacent facts can all become retrieval jobs inside one answer.

It does not follow that publishers should create a page for every possible reformulation. The practical implication is narrower: important evidence should be expressed with enough entity, terminology and context variation to remain retrievable when an agent decomposes the question.

Three judgment sets answer three different questions

The benchmark avoids treating one signal as complete truth. It creates three relevance sets over the same queries and documents:

1. Agent citations: a document counts as relevant when the production agent cited it. This connects retrieval to downstream use, but a redundant useful source can remain uncited once a claim is already supported.
2. Production web ranking: up to 50 results per query from an internal retrieval and reranking system, averaging 43.1 results. This expands coverage but inherits the preferences of that production stack.
3. Combined plus LLM judgments: the union of the first two sets, plus model judgments for previously unlabeled candidates pooled from several retrieval systems.

The combined set averages 99.6 positively judged documents per query. By comparison, the paper lists one judgment per query for MS MARCO Web Search. That depth is central to Q2D-Web’s value, because shallow labels can punish a retriever for finding relevant documents that were never judged.

The lowest-recall system found the largest unique set

BM25 ranked last on overall Recall@1000 in the reported comparison. Yet it recovered the largest set of relevant documents that none of the other tested retrievers found.

That is a useful warning against replacing a retrieval family based on one aggregate score. A dense model can improve semantic matching while losing exact names, rare strings, identifiers or lexical relationships that a sparse system catches. A hybrid system may be valuable because the errors differ, not because every component wins the leaderboard.

For an internal search or RAG evaluation, add a unique-positive report beside the usual mean score:

unique positives for model A
= relevant documents retrieved by A
- relevant documents retrieved by any comparison model

Then inspect samples from that set. A large unique count can represent genuinely complementary evidence, annotation noise or an unintended bias in the relevance pool. The number only becomes actionable after review.

A third of the corpus kept the order but inflated recall

Full-corpus testing is expensive. The paper reports that one pass with its 4-billion-parameter embedding model required 4,608 H200 GPU-hours, while even the 300-million-parameter model required almost 200 H200 GPU-hours.

The authors therefore built smaller evaluation corpora that retain every positive document and select difficult distractors from pooled retrieval runs. Their reciprocal-rank-fusion sample kept about one third of the full corpus and preserved the model ordering under the combined judgments. Absolute Recall@1000 still rose by 3 to 7 points.

That difference is easy to miss. A reduced corpus can be useful for choosing which model to test next while still making every model look better than it does against the full search space. Use sampled-corpus scores for iteration, not as interchangeable replacements for production-scale numbers.

What this means for AI visibility work

• Separate retrieval from citation. A missing citation may begin with candidate generation, reranking, source selection or answer composition. The final answer alone does not locate the failure.
• Test decomposed queries. Include entity lookups, comparison queries, supporting facts and multilingual variants that a search agent might generate.
• Keep lexical evidence visible. Exact product names, model IDs, dates, people, organizations and technical terms can complement semantic prose.
• Measure unique coverage. Compare which relevant sources each retrieval method alone contributes.
• Label benchmark boundaries. A private corpus with production traffic can be realistic without being independently reproducible.

SearchEngineAnswer’s AI citation dataset coverage audit applies the same denominator discipline to citation studies. The AI Search ranking factors guide separates documented platform behavior from inference.

Read the paper and leaderboard

Primary source: Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems, submitted September 8, 2026. The authors also provide a public leaderboard. SearchEngineAnswer did not run the private benchmark.