AI Answer Citations Can Fail at Four Separate Layers

An AI answer can fail before its first sentence is generated. The evidence set may be weak, the correct page may never reach the locator, a summary may break the path back to the source passage, or a final validator may accept a claim that should have been regenerated or refused.

Four recent research papers measure different parts of that chain. Read together, they support a more useful citation audit: inspect each layer separately instead of treating a visible link as proof that retrieval, grounding and attribution all worked.

The failure map has four separate control points

This map extends the citation-selection model with three implementation questions that citation monitoring normally cannot see: whether retrieval was confident enough, whether the evidence page was exposed, and whether an intermediate summary still maps to the original passage.

The PageRecall repository passed its offline tier-0 checks

PageRecall evaluates literature-grounded question answering over a pool of 27,487 research papers and 71 held-out test questions. Its public repository includes predictions, evaluator code, provenance files and a tiered reproduction harness.

SearchEngineAnswer checked repository revision 2fa753ed4aa2f703b76acd6eec7cbfa3b1a60f80 on September 21. Running python3 reproduce.py --tier 0 produced four passes, zero failures and one skip:

• the released validation data matched its pinned SHA-256 hash;
• all 11 committed development metrics matched the paper values within the harness tolerance of 0.0006;
• the organizer validator accepted all 71 test predictions;
• 14 metrics artifacts had provenance blocks, with two named pre-provenance files explicitly exempted;
• the optional check comparing a shipped dist/ archive with the current commit skipped because that archive was absent from the checkout.

Download the reproduction log and boundary note. The result verifies calculations over committed artifacts. It does not independently reproduce the model outputs that created those artifacts.

Page selection can dominate the evidence error

The PageRecall paper reports that its baseline page selector exposed the annotated gold page 52.6% of the time. That is an upstream ceiling for any later component that depends on seeing the relevant page.

The authors also tested what happened after page selection. When the correct page was available, 45 of 48 emitted locators identified the correct page, about 94%. When the page was missing, 24 of 45 cases pointed to the wrong page, 14 returned no locator and only 7 recovered the correct page.

The practical lesson is not that 52.6% applies to web search. The dataset, corpus and pipeline are specialized. The useful lesson is diagnostic: record expected_page and selected_page separately. Otherwise, a fluent answer grounded in the wrong page can look like a citation-formatting problem when the failure happened one stage earlier.

Abstention changes quality by changing what gets answered

Predictable Failure in Multi-Hop Retrieval studies whether score distributions can identify weak evidence sets before generation. Across three multi-hop benchmarks, two retrieval architectures and five reported failure regimes, the paper reports confident-wrong-answer rates from 14.5% to 62.1%.

On MuSiQue with the paper’s LLM-judge architecture, its Retrieval Confidence Score reduced the confident-wrong-answer rate from 39.5% to 20.6% at 50% coverage. That is a reported 47.8% relative reduction, with expected calibration error of 0.035.

The coverage boundary matters. The policy improves the accepted subset by abstaining on the rest. It does not make every question 47.8% more accurate, and the paper is not a live audit of ChatGPT, Gemini, Perplexity or Google Search. A production log should therefore keep confidence, acceptance threshold and coverage beside the result.

Compression can preserve the answer and break the citation

A retrieval system often compresses evidence before generation. The compressed text may be shorter, clearer and sufficient for the answer, but a citation attached to that intermediate text still needs a path back to the publisher’s original passage.

The Attribution-Compression Frontier in Retrieval-Augmented Generation makes that loss measurable. On ASQA, one RECOMP-style setting reported citation precision of 0.86 when citations were judged against generated summaries. After attempting to recover the original source spans, the reported precision was 0.12. The unsupported rate moved from 0.17 against summaries to 0.88 after source recovery.

Those numbers belong to that configuration. The paper uses a shared NLI evaluator without an independent human-calibration study, and its nominal 0.25 budget produced an observed compression ratio of 0.08. Still, the failure distinction is useful: a summary can support the answer while failing to support a verifiable publisher citation.

Final validation needs an action, not just a score

CiteGuard-RAG places a validation layer between retrieval and delivery. Across 400 questions drawn from a controlled housing-law set, PrivacyQA and CUAD, the paper reports 99.1% retrieval accuracy, 98.3% grounded-answer accuracy and 98.3% citation validity in its controlled evaluation.

The external-domain results were harder, particularly for evidence use, span alignment and refusal calibration. That prevents the strongest controlled percentages from becoming a universal claim about open-web answer engines.

The implementation idea is stronger than the headline score: validation should produce one of three actions. Accept when the claim and source span pass. Regenerate when usable evidence exists but the answer or locator fails. Refuse when the evidence set cannot support the claim. A validator that records a low score but still ships the answer has not controlled the failure.

One claim can pass one layer and fail the next

Consider an answer that states, “The policy requires a 30-day notice,” and visibly cites a 40-page document.

1. Retrieval confidence: pass. The policy document is clearly separated from unrelated candidates, so the evidence set is accepted.
2. Page selection: fail. The selector sends pages 2, 4 and 7 to the locator, while the notice rule appears on page 31.
3. Source re-attribution: unknown. A generated summary contains “30-day notice,” but the pipeline cannot map that sentence to page 31 because the page never entered the context.
4. Runtime validation: refuse. The visible document URL is real, but no inspected source span supports the claim. The safe output is a refusal or a new retrieval attempt, not a confident answer with a document-level citation.

This example shows why a domain-level citation count cannot diagnose the system. The final URL can be correct while page exposure and passage support fail.

Use one ledger row per claim and evidence path

The downloadable ledger keeps the four layers in one record without collapsing their decisions. It includes an evidence-set ID, question or claim, retrieval confidence, selected and expected pages, source-span ID, compression transform, attribution status, validation result, abstention decision and final citation.

Download the AI citation failure audit ledger

The two sample rows are marked EXAMPLE-REMOVE. Delete them before use. Do not place private prompts, authentication data, raw account identifiers or confidential source text in a public copy.

Pair this ledger with the claim-level citation support audit when checking whether a passage entails a claim. For high-risk answers, the CiteShade source-deletion method adds a separate causal test. The CITECHOICE analysis explains why citation count and source admission also need different measures.

The publisher decision is to preserve traceability

Publishers cannot inspect every internal retrieval score used by a commercial answer engine. They can make their own evidence easier to preserve: stable URLs, descriptive headings, claim-sized passages, primary-source links, clear table labels and visible update boundaries all reduce ambiguity when a page is retrieved or summarized.

Do not turn that advice into a promise of citation. The four-layer model is a diagnostic contract, not a ranking recipe. The justified measurement question is narrower: when an answer credits a page, can the evidence path be reconstructed from candidate set to page, source span, transform and final validation decision?

Primary research: retrieval confidence and abstention, PageRecall, attribution under compression and CiteGuard-RAG. All four are research papers, not audits of live commercial search products.