An Adobe preprint separates agent use from user success in retrieval measurement

Key finding

LARA proposes links between retrieved evidence and later outcomes. Its reported Adobe Express case had not implemented those attribution fields.

Sonar reconnects a source thread across session doorways to a finished artifact accepted by a user.

A useful source can enter an assistant’s workflow without receiving a visible click. A user can also export a finished artifact in a later session. Those two cases make a session-level click or abandonment number a weak explanation of what retrieval contributed.

Adobe researchers Yan Liang, Anuj Agarwal, and Aditya Reddy Cheruku propose a measurement framework in an October 21 preprint. Called Layered Agentic Retrieval Assessment, or LARA, it separates source retrieval, subsequent use, and product outcomes. The proposal is motivated by an Adobe Express production case.

The proposed attribution fields were not deployed in the reported case

This is the first boundary to keep visible. The paper says its proposed lineage fields were not implemented in that deployment. The reported engagement comparisons rely on timing, and Adobe’s disclosure policy withholds absolute session, user, and query counts. The paper does not demonstrate a measured improvement from deploying LARA.

Its distinction is still useful for designing an audit. A source being selected by an agent is evidence of machine use. A person choosing to continue with the result is a different observation. A completed export concerns the task outcome. Treating all three as the same success event removes the information needed to diagnose a failure.

Follow one artifact across a session boundary

Imagine a fictional design assistant that retrieves a template, places it in a draft, and opens that draft in an editor. The user edits the draft and exports it after leaving the chat. A chat-only report records no export. An editor report records a successful export but lacks the source path.

Before crediting the retrieval, ask whether the export can be linked to the asset that was actually used. If only timestamps connect the events, several intervening retrievals could explain the outcome. Record that uncertainty rather than choosing whichever source happened to appear nearest in time.

Keep the observations distinct
EventWhat it establishesWhat remains unknown
Source returnedA retrieval produced a candidateWhether the candidate entered the artifact
Agent used the sourceThe source entered the machine workflowWhether a person accepted it
User continued or exportedA subsequent action occurredWhich retrieval deserves credit without a durable link
No further signalThe available logs show no observed actionSatisfaction, abandonment and silent completion

Missing retrieval is a separate routing question

The authors also distinguish attribution after a search from a planner failing to search at all. Their proposed lineage cannot measure a missing retrieval event on its own. That question needs a defensible label for whether the original prompt required retrieval.

For a proposed local review, sample prompts independently of whether search ran. Have a reviewer mark the evidence need before inspecting the routing decision. Then examine the retrieved evidence and downstream use only for the relevant cases. This avoids scoring only the calls that happened to produce logs.

These are suggested checks, not SearchEngineAnswer experiments. The production example is a creative assistant; it cannot establish a general ChatGPT referral rate or Google click trend. The paper is submitted to a conference industry track and should be read as a preprint.

Use the AI visibility measurement crosswalk for the differences between platform reports. The research-trace guide covers a separate evidence source: operational traces. For the workflow described here, first establish whether the same source can be followed into the finished artifact; a dashboard total cannot supply a missing link.

Keep learning

Continue this topic

Community discussion

Discuss: An Adobe preprint separates agent use from user success in retrieval measurement

Have a question, a useful example, or a different perspective? Join the discussion, share evidence, and help other readers reach a better answer.

0 replies Moderated
No replies yet.

Be the first to ask a focused question, share a practical example, or add useful evidence.

Ask a question or join the discussion

Share evidence, a useful example, or a clear question. Be specific, stay on topic, and challenge ideas without attacking people. First-time replies may be held for moderation.