How to Test Tavily’s Language Boosting and Strict Filtering

Two controls make two different claims

Tavily’s Search API reference says the language parameter boosts results in the specified language. It accepts an ISO 639-1 code or an English language name, and Tavily recommends writing the query in the same language.

Tavily’s September 2026 product update introduced filter_by_language. When it is true, a language value is required and results that do not match are excluded. That is a compliance rule, not a stronger version of boosting.

Use the 12-row language-control fixture

Download the Tavily language-control test fixture. It contains matched baseline, boost and strict-filter requests for English, Spanish, French and German. Every row is marked EXAMPLE-REMOVE because no request result is included.

Each row preserves the query, query language, parameter value, strict-filter state, search depth, result limit and the fields an evaluator must complete. Keep the query meaning aligned across languages. A literal translation is not always the same information need, so have a fluent reviewer confirm the prompt before running the set.

Use a fixed account, region, date window, search depth and result limit for the matched comparison. If another parameter changes between rows, you no longer know whether the language control caused the difference.

Classify the result before calculating a rate

Do not infer language from the country-code domain. Open the result and classify the title, visible article language and any mixed-language condition. Record redirects and inaccessible pages separately.

Language-identification libraries can speed the review, but short titles, proper nouns and code-heavy pages often need a human decision. Preserve the detector result and the final reviewer decision as separate fields if you automate the first pass.

Calculate three comparisons

Language compliance is the number of reviewed matching-language results divided by all reviewed results. Apply it to every mode, but use it as a hard pass-or-fail gate only for strict filtering.

Domain overlap compares the unique domains returned for baseline, boost and strict filtering. A Jaccard score can show how much the source set changed: intersection divided by union. Report the domain lists as well as the score, because a small sample can make one URL look disproportionately important.

Result availability records how many usable results remain after filtering. Perfect language compliance with very few relevant results may be a poor retrieval outcome. Relevance still needs a separate human judgment.

Keep Safe Search and speed modes separate

Tavily documents safe_search as applying to both web and image results. It is not supported with fast or ultra-fast search depth. Do not add Safe Search to only one language row or silently switch depth to make a request pass.

If safety behavior matters, run a second fixture where Safe Search is the independent variable and use a supported depth for every row. The language fixture should remain focused on language selection.

Set decision rules before the run

1. Fail a strict-filter row when a reviewer confirms one nonmatching result.
2. Do not fail a boost row merely because a foreign-language result appears.
3. Flag any row with more than 20 percent unknown results for rerun or manual inspection.
4. Report relevance separately from language compliance.
5. Repeat the fixture on another date before treating a difference as stable.

This guide does not report Tavily performance. It turns the documented parameter contract into a repeatable evaluation that can produce a defensible result. For a broader experiment design, use the small SEO experiment method. For source-level reporting boundaries, see the AI answer citation failure layers.