ChatGPT Referral Growth: Separate AEO Lift from Platform Tailwind

Read the study as a method, not a benchmark

Watanabe and Nakayashiki studied one high-traffic domain, glasp.co. The treated group was a large collection of YouTube question-and-answer pages that received a bundle of AEO changes in January 2026. Other paths on the same domain formed the untreated control. Both groups shared the domain, analytics configuration, bot-filtering regime, and much of the platform tailwind.

The paper used first-party Google Analytics 4 data for ChatGPT referral sessions and Google Search Console for organic-search guardrails. It did not publish absolute session counts; it reported indexed series and relative changes. That protects commercially sensitive volumes but limits independent reconstruction of the complete result.

Define the decision before exporting data

Decide what the analysis must support. A content team may need to choose whether to expand an intervention, hold it, revise it, or stop it. That is narrower than proving that “AEO works.” Write the treatment, affected URL set, primary outcome, guardrails, observation window, and decision threshold before looking at the post-change graph.

Freeze the treatment membership. Moving successful pages into the treated group after the fact creates selection bias. Keep removed URLs, zero days, failed tracking periods, and changed templates visible. If the intervention is a bundle, say so; the study cannot isolate which component produced the estimated change.

Choose a control that shares the tailwind

An on-domain control can share brand demand, analytics, infrastructure, and platform-level growth. It should also resemble the treated group in reader job, page age, template, baseline referral level, and organic-search exposure. A sitewide “everything else” control is easy to define but can hide large content differences.

If no credible on-domain control exists, consider a staggered rollout, matched URL groups, or a longer pre-period. Do not replace a weak control with a historical average and then use causal language. A before/after comparison can still be useful as a descriptive operating metric when its limitations are explicit.

Preserve the referral definition

OpenAI currently says ChatGPT referral links include utm_source=chatgpt.com. Google Analytics defines session source as the source that initiated a session, and its Data API exposes dimensions such as sessionSource and landingPagePlusQueryString. Record the exact dimension, filter expression, property, time zone, consent configuration, and export date.

Do not silently merge referral sessions, engaged sessions, server requests, citations, and conversions. Analytics can miss visits because of consent, browser behavior, redirects, or collection failures. Server logs observe requests but do not identify every human or business outcome. Use both as complementary evidence and preserve the mismatch.

Calculate a descriptive ratio before a model

A transparent first check is the ratio of treated growth to control growth. In the paper, the monthly treated series grew 6.1× from January to May while the control grew about 3.5×, producing a simple ratio near 1.75×. The paper’s 5.7× figure describes total ChatGPT referral growth, so dividing 5.7 by 3.5 answers a different descriptive question and yields about 1.63×.

Neither calculation replaces the interrupted time-series model. The model uses weekly treated/control ratios, the intervention boundary, the pre-existing trend, and autocorrelation-robust inference. Report the simple arithmetic because readers can reproduce it, then report the model as a separate result with its assumptions.

Keep placebo and sensitivity results visible

The study found a statistically significant 1.82× level shift under heteroskedasticity-and-autocorrelation-consistent errors, yet its placebo-in-time permutation test did not pass the conventional threshold. That is not a contradiction to hide. The two checks ask whether the observed break looks persuasive under different assumptions.

Run placebo dates, alternative pre-windows, engagement-filtered outcomes, and spike exclusions when the sample supports them. Predeclare which check can downgrade the decision. A positive central estimate followed by a failed falsification test should lead to “hold and collect more data” more often than “scale because the chart went up.”

Protect organic search with separate guardrails

Use Search Console clicks and impressions for the treated and comparison URLs as guardrails, not as components of one blended “visibility” score. Google’s Search Analytics API can group and filter by page and date, but it returns top rows rather than guaranteeing every row. Save request bodies, property scope, aggregation type, data state, and omitted dates.

The paper reports that treated organic clicks did not fall beyond the surrounding site trend and that indexation was preserved in its setting. That observation does not guarantee safety for another rewrite. Define a rollback threshold for organic clicks, index coverage, conversions, and content-quality failures before launch.

Download the tailwind-adjustment ledger

The CSV below connects the treatment definition, control group, analytics and log rules, raw outcomes, descriptive ratio, model result, placebo test, concurrent changes, and final decision. Its sample rows begin with EXAMPLE-REMOVE; delete them before recording real observations.

Download the AEO referral tailwind-adjustment ledger (CSV)

Keep the ledger beside versioned exports and analysis code. The AI referral workflow helps preserve analytics definitions, while the AI visibility measurement crosswalk keeps mentions, citations, visits, and outcomes separate.

Common reporting failures

• Using total channel growth as treatment lift: the platform tailwind remains inside the number.
• Choosing controls after seeing outcomes: this lets the desired conclusion shape the denominator.
• Removing zeros and tracking breaks: the cleaned chart becomes less representative than the operating system.
• Calling a preprint a universal benchmark: the study covers one domain, one intervention bundle, and a ChatGPT-dominant channel.
• Reporting only a significant model: the failed placebo test and short pre-period materially qualify the conclusion.
• Ignoring organic guardrails: referral gain is not a win if the same release damages a more valuable channel.

Limits and re-audit triggers

An observational treated/control design cannot remove every confounder. Spillover can lift the control, templates can respond differently to the same release, and analytics filters can change the composition of recorded sessions. Absolute volumes are withheld in the source study, and the public result is still a preprint.

Start a new analysis when the treatment bundle, group membership, ChatGPT referral tagging, GA4 property, consent mode, bot filter, Search Console property, site template, or platform surface changes. Preserve the old run instead of overwriting it. The honest output may be “insufficient evidence”; that is a useful decision when the alternative is attributing a platform-wide surge to one editorial change.

Primary sources

• Watanabe and Nakayashiki: Disentangling Answer Engine Optimization from Platform Growth ; study design, reported results, and threats to validity.
• Google Analytics Data API dimensions and metrics ; session-source and landing-page field definitions.
• Google Search Analytics API ; request fields, aggregation, and row-limit boundaries.
• OpenAI publishers and developers FAQ ; current ChatGPT referral-tag guidance.

Source check: August 31, 2026. Reopen product documentation and preserve the paper version used for each analysis.