Mistral Search Toolkit: Evaluate Retrieval Before Blaming the LLM
Mistral Search Toolkit supports lexical, vector, and hybrid retrieval with reranking. Evaluate ingestion and retrieval before scoring the generated answer.
Direct answer: Evaluate Mistral Search Toolkit as a retrieval pipeline before judging the language model that consumes its results. Mistral documents document ingestion, lexical BM25 retrieval, vector retrieval, hybrid retrieval, query preprocessing, reranking, and common information-retrieval metrics.
A weak answer can begin with a missing document, a parsing error, the wrong chunk, a vocabulary mismatch, poor fusion, or a reranker that demotes the decisive passage. Scoring only the final prose hides the repairable layer.
Freeze the corpus and ingestion record
Create a versioned corpus manifest with document identifiers, source URLs or file paths, owners, dates, content hashes, permissions, MIME types, languages, and expected parsing outcomes. Preserve rejected documents and ingestion errors. A benchmark that silently drops hard files can overstate retrieval quality.
For representative documents, compare extracted text with the source. Check headings, tables, lists, links, page boundaries, metadata, repeated navigation, and scanned content. Record chunk identifiers and the source offsets needed to reconstruct a returned passage.
Define a query set and relevance judgments
| Query class | Purpose | Expected challenge |
|---|---|---|
| Exact term | Test lexical matching | Tokenization and field choice |
| Paraphrase | Test semantic matching | Vocabulary mismatch |
| Identifier | Test codes and names | Embedding weakness |
| Multi-constraint | Test complete passage relevance | Partial matches |
| No-answer | Test abstention evidence | Irrelevant nearest neighbors |
Label relevance before comparing configurations. Use more than one assessor for ambiguous tasks, preserve disagreements, and state whether the judgment applies to a document, chunk, passage, or answer.
Compare BM25, vector, and hybrid retrieval
Run the same frozen queries against each mode. Keep corpus, top-k, filters, preprocessing, and evaluation judgments constant. Save the ranked identifiers and scores before reranking.
BM25 can excel when exact terms, names, and identifiers matter. Vector retrieval can recover semantically related language. Hybrid retrieval can combine evidence, but its fusion settings and candidate pools must be recorded. “Hybrid” is not one reproducible configuration.
Inspect failures, not just averages. A mode with a strong mean can still miss every query in a critical product class. Report results by query class and preserve no-result and error cases.
Measure preprocessing and reranking separately
Query rewriting or preprocessing can improve recall, but it can also remove an identifier, change an entity, or broaden the task. Store the original and processed query together. Compare retrieval before and after preprocessing on the same judgments.
For reranking, save the candidate set, original order, reranked order, scores, and model or configuration version. Measure whether relevant passages move upward and whether decisive passages disappear from the final context window.
Use precision and recall at a declared cutoff, MRR for first-relevant-result tasks, and NDCG when graded relevance and ordering matter. Define the formula and denominator rather than publishing one unexplained composite score.
Connect retrieval to generation without collapsing them
After the retrieval study, freeze the selected evidence and evaluate generation separately for claim support, completeness, boundaries, and abstention. Keep the prompt, model, context ordering, truncation, and output. A generation change should not alter the retrieval benchmark unnoticed.
SearchEngineAnswer has not run a comparative Mistral Search Toolkit benchmark for this article. The toolkit is in public preview, and behavior can change. This is a reproducible method, not a performance ranking.
Use the tool-score evaluation guide for transparent metrics and the evidence-led publishing guide for the final claim audit.
Ask a question or join the discussion