ChatGPT, Claude, Grok and DeepSeek Search the Web Differently

A study of 171,264 real conversations and a controlled replay of 1,000 prompts shows that ChatGPT, Claude, Grok and DeepSeek do not share one web-search behavior. They decide to search at different rates, formulate queries differently, receive different result pools and sometimes use retrieved material without citing it.

The study at a glance

The observed dataset contained 48,703 turns with web search. ChatGPT contributed most users and conversations, so raw totals should not be used as a platform popularity comparison. The controlled replay is more useful for comparing model behavior because each system received the same prompt set.

The largest difference appears before a result exists

In the controlled replay, GPT-5.3-chat searched for 140 of the 1,000 prompts. Claude Sonnet 4.6 searched for 825, Grok-4.3 for 766 and DeepSeek-v4-flash for 584. That is a sixfold gap between the most conservative and most search-active systems.

More search was not automatically better. The paper reports that enabling search generally helped GPT-5.3-chat, Grok-4.3 and DeepSeek-v4-flash on prompts where those models chose to search. Claude searched far more often, but its measured effects were mixed. The models also scored higher on prompts they chose not to search, which suggests that recognizing an answerable question can be easier than repairing a real knowledge gap with retrieval.

The researchers changed publicly available search instructions across three OpenAI models and found that the instruction harness affected call rates. The underlying model still had a larger effect. For practitioners, that means the visible product name is not enough context for a benchmark. The model, system instructions, tool contract and product surface all belong in the test record.

The agents compress prompts, then diverge in strategy

Nearly 80% of the observed user prompts contained more than 20 terms, while almost all generated web queries stayed below 10 to 15 terms. The agents were not simply copying prompts into a search box. They extracted shorter queries from the current message, earlier conversation context and prior search results.

The observed products also used different search shapes. ChatGPT, Grok and DeepSeek issued parallel groups of queries, while Claude searched one query per iteration and went deeper across successive steps. In the controlled API setting, only GPT-5.3 used parallel fan-out. This product-versus-API difference is a warning against presenting one API trace as a complete description of the consumer assistant.

Result pools were small and platform-specific

Those result pools are not neutral samples of the web. The authors found strong domain preferences in each platform’s returned results and final citations. A domain can therefore lose citations because it was not retrieved, because it was retrieved at an uncompetitive rank, or because it was retrieved and used but omitted from the visible citation list. Those are three different problems.

A second dataset makes the search gate measurable

A September 13 dataset from Prefer held 80 AI-search questions constant across three fresh runs of ChatGPT, Perplexity, Gemini and Claude. Web search was requested on all 960 API calls, but the engines reported search activation on only 861 answers.

All 99 answers that skipped search also lacked citations. Once search ran, 858 of 861 answers returned at least one source. This does not prove that every consumer product behaves the same way, because the test used APIs and a topic-specific prompt set. It does show why a missing citation can originate before domain selection. The full cross-engine source-market analysis separates that gate from source volume and platform preference.

A visible citation list does not reveal every source used

The study’s claim-level analysis found that a non-trivial share of claims could be attributed to retrieved URLs that were not cited in the answer. Depending on platform and setting, the paper reports rates from roughly 14% to 53%. This is not proof that every matching phrase was copied from the uncited page. It is evidence that citation selection can omit retrieved support that appears relevant to a claim.

The URL audit adds another distinction. In the observed dataset, 85.99% of ChatGPT’s retrieved citations, 88.39% of Claude’s, 91.82% of Grok’s and 88.17% of DeepSeek’s were classified as valid. The remainder included unknown, hallucinated or dead URLs. A citation count that ignores reachability and support quality can reward a broken or unverifiable reference.

Use a five-stage AI-search audit

1. Search invocation: Did the system search, and was the question likely to need fresh external evidence?
2. Query formulation: What queries were issued, in what order, and which terms came from earlier results?
3. Retrieved domains: Which URLs entered the candidate pool, at what positions, and were they reachable?
4. Citation selection: Which retrieved pages became visible citations, and which were omitted?
5. Claim support: Does each important claim follow from the cited page, an uncited retrieved page, or neither?

This lifecycle also changes how publishers interpret visibility. Improving a title may help a candidate page, but it cannot fix a system that never searches for the prompt or a retrieval layer that does not return the domain. Likewise, winning a citation does not prove that the answer used the page accurately.

Download the lifecycle worksheet

The worksheet keeps call decisions, queries, candidate URLs, citations and claim support in one row-level record. Its first line is marked EXAMPLE-REMOVE so it cannot be confused with measured results.

Download the agentic-search lifecycle worksheet

My judgment for publishers

The paper is strongest as a map of where visibility can be won or lost, not as a permanent league table. The model identifiers and product harnesses will change. The more durable insight is that AI search is a chain of decisions with several hidden filters. A serious monitoring program should preserve prompts and dates, record the search trace when available, validate citations, and repeat the same protocol across product surfaces.

This also argues against one-number “AI visibility” scores. A single share can blend search propensity, query rewriting, source retrieval, citation choice and answer synthesis into a metric that is easy to compare but hard to diagnose. Use the AI crawler guide for access controls and the AI SEO framework for the broader measurement system.

What this study cannot establish

• The donated conversations are opt-in and mostly English, so they do not represent every user or market.
• The raw conversations cannot be released, although the authors provide analysis code and mock data.
• Platform interfaces and APIs can use different search harnesses, as the study itself demonstrates.
• Several evaluations use platform-specific LLM judges. Human checks found more than 90% agreement for query specificity and 67% to 77% within-one agreement for response-quality scores, but automated judgment remains a measurement choice.
• The paper is a September 16 preprint, not a peer-reviewed final publication.

Primary sources

• Characterizing Web Search by Conversational LLM Agents, arXiv preprint, September 16, 2026.
• AgenticSearchLens code and mock-data repository, reviewed September 20, 2026.