Evaluation & operations · Explore this field ↗ · Implementation · 4 min read
Model-change evaluation: compare a changed system without a live leaderboard
When a model, configuration, or system prompt changes, hold the other inputs steady, compare paired cases, and inspect the slices that can be harmed by an average improvement.
Paired comparison flow
- 01Freeze system inputs
- 02Run baseline and candidate
- 03Blind-score properties
- 04Inspect regressions
- 05Record bounded decision
The output is a release decision, not a live vendor leaderboard.
Original conceptual diagram · not a live trace or measured result.Name the change precisely
“We upgraded the model” is not an evaluable change description. Record model identifier and provider configuration where available, system instructions, temperature or comparable settings, tool definitions, retrieval configuration, knowledge snapshot, locale, channel, test runner, and date. If any of those vary between baseline and candidate, classify the run as a system comparison and say so. A configuration record also gives a later incident investigation somewhere concrete to start.
NIST’s AI RMF describes risk management across the lifecycle, including evaluation. For a product team, that means a model setting cannot be separated from the system around it by wishful naming. Freeze what you can, document what you cannot, and keep the claim as narrow as the experiment. A paired result can support a bounded release choice; it does not prove an abstract quality ranking or permanent capability gain.
Build paired cases with fixed inputs
Choose cases that represent the release decision: routine requests, expensive mistakes, refusals, retrieval questions, actions with mocked responses, corrections, and handoff. Run the same inputs through baseline and candidate using the same authorized test state, knowledge snapshot, clock, and tool stubs. Randomize reviewer order and hide which output came from which configuration when practical. Keep test data synthetic or properly minimized, especially when tool traces include account context.
For generated language, compare properties rather than a single ideal paragraph. Did it state the required limit? Did it cite the approved source? Did it ask the required question before an action? Did the stub confirm before it claimed success? Preserve the full trace necessary to answer those questions. For deterministic flow choices, compare state transitions and typed arguments directly; do not let subjective prose preference decide an authorization result.
Predeclare slices and decisions
Define cohorts before running: action versus information tasks, language, channel, account state, accessibility route, risk tier, and known hard cases. Set a small decision table: blocked if a must-not-fail case regresses; hold if an important cohort is inconclusive; advance only if required checks pass and reviewers can explain meaningful tradeoffs. Do not create a public leaderboard from this internal comparison; the service contract, fixtures, and traffic are not a neutral test of every use case.
Worked example, hypothetical: a candidate writes shorter answers and improves average reviewer preference, but two cancellation fixtures omit the mandatory confirmation. The result is blocked, even if an overall average rises. The correct action is to inspect the instruction, tool contract, and model interaction—not to bury the regression in a composite score. An average is a summary, never permission to violate a non-negotiable action boundary.
Inspect divergence, not only totals
For every paired disagreement, compare route, retrieved evidence, tool call, final claim, and repair behavior. Cluster changes by failure type: lost source constraint, new unsupported detail, different tool selection, unsafe refusal, or better clarification. Keep a few representative examples beside counts so reviewers understand the mechanism rather than treating statistics as a verdict. Record whether a difference is intended product change, environment variance, judge uncertainty, or candidate regression.
A live traffic comparison can be useful after a controlled release, but it is not a substitute for paired evaluation. Google SRE notes that changes in time and shared dependencies can confound comparisons. Production differences may reflect seasonality, customer mix, outages, or a parallel policy update rather than the model configuration. Preserve a stable offline suite so that a later production signal has a reference point with fewer moving parts.
Limits and acceptance criteria
Accept a candidate only when all gated cases meet the declared property checks, reviewers have adjudicated material disagreements, and the release record explains known regressions and mitigations. Keep the baseline available for rollback and rerun a focused suite after any prompt, tool, or source change; the result is valid only for the frozen package tested. Include a negative test that should remain refused or inert, because capability changes often widen behavior unintentionally.
This process does not establish general intelligence, vendor ranking, or durable superiority. Small suites miss unknown inputs, reviewers can disagree, and nondeterministic outputs may vary across runs. The value is narrower and more useful: a defensible decision about one proposed configuration for one bounded service. If live evidence later disagrees, reopen the decision with its original assumptions rather than claiming the offline test was a forecast.
Take it into the review
Paired model-change record
| Case slice | Frozen inputs | Baseline result | Candidate result | Decision |
|---|---|---|---|---|
| Cancellation action | Prompt, tool stub, policy v19 | Confirmation requested | Confirmation omitted | Block |
| FAQ retrieval | Index v19, locale en | Approved source cited | Approved source cited | Pass |
| Repair | Corrected date turn | State cleared | State cleared | Pass |
A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.
What the sources establish
AI Risk Management Framework
NIST states the framework is intended to incorporate trustworthiness considerations into AI system design, development, use, and evaluation.
Limits: It does not prescribe paired-test implementation details or model rankings.
Checked 2026-09-19 · NIST · source publication date not established.
Open original source ↗Canarying Releases
Google SRE cautions that time, shared dependencies, and imperfect isolation can confound release comparisons.
Limits: Its canary discussion is not a model-evaluation protocol.
Checked 2026-09-19 · Google SRE · source publication date not established.
Open original source ↗Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.