Evaluation & operations · Explore this field ↗ · Operations · 4 min read
Release gates and rollback: make stopping a change routine
A release package should say exactly what changed, what will be observed, who can stop it, and how to return to a known safe configuration without inventing a universal threshold.
Staged release flow
- 01Package change
- 02Run pinned checks
- 03Limited exposure
- 04Decide from signals
- 05Proceed or restore
Stop conditions trigger investigation; they do not alone prove causation.
Original conceptual diagram · not a live trace or measured result.Release the whole behavior package
A conversational release is more than application code. It may include a model setting, system instruction, retrieval index, policy text, tool schema, feature flag, channel UI, and a changed handoff queue. List every changed component and its predecessor in one release artifact. Include the evaluation suite version, knowledge snapshot, rollout audience, owner, rollback target, and known assumptions. If a dependency cannot be pinned, state that risk rather than implying a reproducible release.
Google’s SRE guidance describes a canary as a partial, time-limited deployment paired with an evaluation process and integrated release decision. The transferable lesson is not a prescribed percentage or duration. It is that a release needs a defined comparison and a way to stop, rather than a hopeful observation after everyone has already received it. Separate release notes should distinguish a source refresh from a model change, because their failure patterns and rollback methods differ.
Set gates before looking at results
Use three kinds of gates. Pre-release gates verify a reviewed artifact, test fixtures, required approvals, and a usable rollback. Exposure gates watch the small cohort for declared safety, action, and service-continuity signals. Stop gates identify conditions that require a pause and investigation, even if the change may not ultimately be the cause. Write these before exposure, not after a troubling dashboard movement.
Write conditions in terms of observable events, not manufactured benchmarks: any unauthorized action observed; required refusal route unavailable; a sentinel retrieval question cites a withdrawn source; a human handoff packet fails to arrive; or the control path becomes unavailable. Name who may stop the rollout and how they notify the incident or service owner. A stop is not a failure of the operator; it is the gate doing its job.
Stage the change and preserve a control
Begin with a deterministic pre-production replay using pinned sources and tool stubs. Then expose the smallest appropriate cohort or internal route, keeping a known working configuration available. Segment observation by task and consequence so a healthy FAQ path does not hide a broken cancellation path. Avoid simultaneous unrelated releases when diagnosis matters. Record any shared infrastructure that can blur baseline and candidate behavior.
Worked example, hypothetical: a team changes both the refund policy index and the model. Their release artifact refuses to bundle them. It first promotes the index with model settings fixed, tests sentinel questions and a limited support cohort, then evaluates the model change separately. The counterexample is a single “assistant upgrade” switch that makes incorrect answers, tool changes, and UI edits indistinguishable. The goal is not bureaucracy; it is a smaller, legible rollback surface.
Rollback is a product capability
A rollback target must be available before promotion. That can mean a prior prompt and model configuration, a previous index alias, a disabled tool, a safe static response, or a handoff-only route. Record the exact restoration action and any state that cannot be reversed. Google SRE warns that rollback is easier when configuration is hermetic; a configuration that depends on mutable external material may not restore the behavior reviewers saw. Rehearse the operator permissions and expected user message.
Do not treat rollback as a destructive undo recipe. If a wrong action has changed an external record, stop further automation and use the system of record’s approved reconciliation procedure. The rollback itself restores future routing; it may not repair past effects. Include the data owner in the gate whenever a release can create commitments, alter eligibility, or expose protected information.
Review the decision, including uncertainty
At each stage, capture the decision, evidence observed, exposure description, missing signals, and next action: proceed, hold, roll back, or switch to manual handling. A stop condition is not proof that the candidate caused the problem. Canary guidance notes that shared dependencies and imperfect isolation can muddy attribution. Preserve that uncertainty while taking the safe action; diagnosis can continue after impact is contained.
Success is a team that can demonstrate a clean stop and a return to the safe route in a rehearsal. Do not claim a release is safe because it passed a small cohort. Staging reduces blast radius and improves diagnosis; it cannot prove every language, account state, or external dependency will behave correctly. A written gate record is valuable precisely when reviewers later need to explain why they released, paused, or restored a route.
Take it into the review
Release decision matrix
| Component | Version change | Pre-check | Stop condition | Rollback target |
|---|---|---|---|---|
| Knowledge index | Policy bundle 18 to 19 | Sentinel citations | Withdrawn source returned | Index alias 18 |
| Tool schema | Address update v2 | Stubbed action replay | Unconfirmed change claim | Disable v2 route |
| Model config | Pinned settings change | Paired evaluation | Safety case regression | Prior settings ID |
A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.
What the sources establish
Canarying Releases
Google SRE describes canarying as a partial and time-limited deployment with an evaluation process integrated into release decisions.
Limits: Its examples and traffic assumptions are not a universal rollout threshold for conversational systems.
Checked 2026-09-19 · Google SRE · source publication date not established.
Open original source ↗Configuration Design and Best Practices
Google SRE notes rollback is important for reducing incident duration and that non-hermetic configuration can be hard to roll back.
Limits: The source does not define a chatbot release artifact.
Checked 2026-09-19 · Google SRE · source publication date not established.
Open original source ↗Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.