Kagi’s AI-Off Switch: A Controlled Search Comparison Protocol
Kagi lets users disable AI features in Search. This protocol holds account settings, queries, lenses, domain rules, and environment constant without inventing a result.
Protocol published August 10, 2026: Kagi’s July 2 changelog says users can completely turn off AI-based features in Search through the AI settings. That confirms the control exists. It does not establish how a particular result set, latency, layout, or user decision changes when the control is switched.
This protocol compares AI-on and AI-off modes while holding the query, account profile, lenses, domain rules, location, language, date, and result depth as constant as possible. SearchEngineAnswer has not run the controlled sample, so no outcome is reported here.
Define the question before the query set
“Is AI better?” is too broad. Choose one falsifiable question: does disabling AI change the visible result modules, the ordered organic URLs, the time to a usable result, the number of source domains, or the rate at which a human assessor completes a declared task?
The control can affect presentation or supporting features without changing every traditional result. Record the complete page rather than only the first ten links, and classify modules before comparing them.
| Variable | Control method | Why it matters |
|---|---|---|
| Account profile | Use one dedicated account and save a safe configuration snapshot | Account settings can shape results |
| Lenses and domain rules | Freeze lenses, upranks, downranks, and blocklists | They can change retrieval independently of the AI switch |
| Query | Use a versioned fixed set | Paraphrases can produce different results |
| Environment | Record date, time, region, language, device, and login state | Search systems and indexes change |
| AI setting | Change only the documented control between paired runs | The comparison needs one material difference |
Build a query sample
Stratify the sample instead of collecting convenient examples. Include navigational, current-fact, explanatory, comparison, local, shopping, troubleshooting, and ambiguous queries if those jobs match the intended use. Define inclusion rules before observing results.
- Create a query-set version and hash.
- Randomize whether each pair starts with AI on or off to reduce order effects.
- Wait a declared interval between paired runs and record unavoidable index changes.
- Save full-page screenshots or permitted exports, not cropped winners.
- Retain zero-result, error, and unexpected-module cases.
- Blind the human assessor to mode where the interface allows it.
A small sample can identify reproducible differences inside that sample. It cannot establish market-wide quality or a permanent Kagi behavior.
Score layers separately
- Presentation: modules shown, answer blocks, summaries, controls, and visual footprint.
- Retrieval: URL overlap, ordering, unique domains, freshness date, and duplicate results.
- Evidence: source visibility, source-owner match, and whether a claim can be traced.
- Task completion: predefined success, assessor time, errors, and confidence.
- Performance: response time under the same client and network conditions.
Do not combine those measures into a single score without publishing the weights. An interface can be quieter while retrieval stays similar, or a summary can be convenient while obscuring evidence. Use the tool-score evaluation guide to expose each layer.
The prior Kagi API reproducibility guide explains why the account behind a key or session is part of the method. The same principle applies to this consumer-search comparison.
Publish the run manifest
{
"query_set": "kagi-ai-switch-v1",
"account_profile": "controlled-v1",
"ai_mode": "on-or-off",
"lens": "none",
"domain_rules_version": "2026-08-10",
"locale": "en-US",
"result_depth": 20,
"run_time_utc": "record-per-query"
}Publish the query set, configuration fields, rubric, exclusions, raw captures where permitted, and calculation. Keep the account credential secret. A configuration label and hash should connect runs without exposing personal settings or authentication data.
Report the release state too. Kagi says it plans to add the option to onboarding; a plan is not a shipped onboarding behavior until the changelog or product confirms it.
Ask a question or join the discussion