Kagi Search API Personalization: Make Retrieval Tests Reproducible

Why the API key is part of the method

Many test reports record the query, endpoint, and date but treat the credential as a neutral secret. Kagi’s announcement means the account behind the key can influence the result set. Two valid keys may produce different results when their lenses, domain upranks, downranks, or blocklists differ.

The key itself should remain secret. The reproducible unit is a safe configuration snapshot: account tier, endpoint, active lens, relevant domain rules, location or language settings, page size, and request parameters. Hash or label the credential so repeated runs can be linked without exposing it.

Define three test modes

Production mode uses the same account configuration as the application. It answers whether the live system behaves as expected, but it can change when operators edit preferences.

Controlled mode uses a dedicated account with a documented, frozen preference set. It is better for repeated evaluations and regression tests. Changes require a versioned configuration record.

Personalization comparison mode holds the query and request parameters constant while changing one preference, such as a lens or domain downrank. It can show an association between that setting and the returned results within the sample; it does not establish a general relevance advantage.

Do not mix the modes in one chart. A production observation and a controlled benchmark answer different questions.

Create a run manifest

{
  "run_id": "2026-08-09-a",
  "endpoint": "search",
  "account_profile": "controlled-v1",
  "lens": "none",
  "domain_rules_version": "2026-08-09",
  "locale": "en-US",
  "query_set_hash": "…",
  "request_parameters": {"limit": 10}
}

Save the manifest beside the raw response and evaluation output. Include error responses, timeouts, and zero-result queries. Removing failures makes an API appear more reliable than the actual workflow.

Version the query set and evaluation rubric. If a result is scored for freshness, authority, diversity, or task completion, define each criterion before reviewing the output and preserve the assessor’s notes.

Prevent configuration drift

1. Use a dedicated API account for controlled evaluations.
2. Export or manually record all relevant preferences before the first run.
3. Restrict who can edit the profile and log approved changes.
4. Run a small sentinel query set before each benchmark.
5. Stop the run when sentinel results or configuration hashes change unexpectedly.
6. Create a new profile version instead of silently overwriting the old one.

Kagi’s product can evolve during a study, and a public preview may change. Record the changelog state and test date. Re-running old queries months later is a new observation, not a continuation of an unchanged instrument.

Report results with the correct boundary

A valid report can say that one documented profile returned a particular set of results for a fixed query set on a stated date. It should not claim “Kagi ranks this domain” without describing the account preferences that may have shaped the result.

If the API powers an AI answer system, preserve the retrieval response separately from the model prompt and answer. Search quality, context assembly, model reasoning, citation choice, and final wording are different layers. The tool-score evaluation guide provides a rubric for avoiding composite scores that hide those distinctions.

Primary announcement

• Kagi Search changelog: Search API public preview and account preferences