chat-latest Is a Moving Target: Freeze Model IDs for Reproducible Tests
OpenAI says chat-latest is regularly updated. Use a fixed model for regression tests and a complete run manifest when comparing current ChatGPT-aligned behavior.
Direct answer: Do not use chat-latest as the only model identifier in a reproducible evaluation. OpenAI describes it as a regularly updated snapshot aligned with the latest model available in ChatGPT for Plus and Pro users. That makes it useful for testing current chat behavior, but unsuitable as a frozen instrument unless every run records the returned model context and date.
This guide defines a protocol. It does not publish a benchmark result or claim that a moving alias is better or worse than a fixed production model.
Choose the model for the question
Use a moving alias when the question is, “How does the current ChatGPT-aligned model handle this task today?” Use a fixed model identifier when the question is, “Did our prompt, retrieval system or application change?” Those are different experiments.
| Test objective | Preferred identifier | Main limitation |
|---|---|---|
| Current experience watch | chat-latest | Underlying snapshot can change |
| Regression test | Fixed production model | May not match current ChatGPT |
| Migration comparison | Both, saved separately | More variables and cost |
| Published benchmark | Fixed ID plus complete manifest | Finding remains time-bounded |
OpenAI recommends GPT-5.6 Sol for production API use in the current changelog entry. Treat that recommendation as product documentation, not independent evidence that it fits every workload.
Freeze the rest of the run
Version the system and developer instructions, user prompt, input documents, retrieval corpus, tool definitions, search setting, temperature or other sampling controls, output schema, SDK, endpoint, region, account, evaluator and rubric. Save the time zone and exact request time.
For search or citation work, preserve whether search was invoked, the queries or tool calls exposed by the response, cited URLs, citation spans, other source links and the visible answer. A citation difference can come from retrieval or rendering rather than the base model alone.
Do not include private prompts, credentials or licensed source text in a public artifact. Hash or label sensitive inputs and describe the boundary needed to interpret the result.
Run a paired observation
- Create a small, fixed task set with expected evidence and failure criteria.
- Run the set against
chat-latestand the selected fixed model in the same observation window. - Repeat enough times to expose nondeterministic variation instead of reporting one convenient answer.
- Save raw responses before normalization or grading.
- Score factual support, task completion, citation match and format separately.
- Repeat on a later date with unchanged inputs and a new run identifier.
If the later chat-latest result changes while the fixed model remains stable, the pattern is consistent with a moving instrument. It still does not identify the exact platform change without model-resolution metadata or an OpenAI disclosure.
Report the boundary
A defensible report states that a named alias and fixed model returned specified outputs for a saved task set, account state and date. It does not say “ChatGPT always prefers” a source, brand or answer pattern.
Keep model quality, retrieval quality, citation choice, latency, cost and editorial usefulness as separate fields. A composite score can hide that a faster answer lost evidence or that a well-cited answer failed the requested task.
Use the tool-score evaluation guide for the rubric and the small experiment method for baselines, confounders and stopping rules.
Ask a question or join the discussion