Evaluation & operations · Explore this field ↗ · Practice · 3 min read
Conversation replay and regression testing: turn real failures into controlled fixtures
How to preserve the meaningful shape of a failed conversation, replay it safely, and assert behavior without pretending language systems are deterministic everywhere.
See the source / a related case
Preserve the decision, not the whole transcript
A useful replay fixture contains the smallest sequence that exposes the failure: user turns, initial state, authorized profile or mock service responses, relevant channel metadata, and the outcome that should occur. Remove or replace personal content before the fixture enters general development access. Retain the original incident separately under the applicable access rules if it is needed for investigation. Name the fixture after the behavior—‘change address invalidates quote’—rather than the individual or ticket. The aim is not to simulate every human phrase; it is to keep the decision boundary that a future change must not cross.
Make the environment reproducible
Pin the workflow or policy version, knowledge snapshot, tool stubs, time-dependent inputs, locale, feature flags, and starting state. A replay that queries a live account or a changing document is a monitoring exercise, not a stable regression test. Mock external services with realistic success, timeout, duplicate, and refusal responses. If a test needs a clock, make the clock explicit. Flowise documents step-by-step tracing for Agentflow V2, which can help identify a run’s sequence; use traces as evidence to build fixtures, not as a substitute for a controlled test environment.
Assert events and constraints
For deterministic flows, assert state transitions, selected route, typed tool arguments, confirmation requirement, and final status. For generated text, assert bounded properties: cites an approved source, asks a required clarification, does not claim an action completed before the stub confirms it, or routes to a person. Avoid asserting a single complete paragraph unless exact regulated wording is genuinely required. Rasa’s end-to-end test cases support user steps alongside expected bot events and slot state; its documentation also notes that expected events after a user step may be evaluated by occurrence rather than a rigid order. Choose assertions that describe the real service promise.
Include interruptions and repairs
Happy paths miss the conversations that most often regress. Build fixtures for a correction after collection, a second message before the first response, unsupported intent during a task, expired authorization, failed retrieval, user cancellation, and handoff followed by a return. For actions, test idempotency with a replay of the same submission and verify the visible status before a retry. A voice or embodied surface also needs its own fixture for barge-in and cancelled output, but do not conflate presentation timing with the underlying action result. The replay should state exactly which layer it exercises.
Triage failures before updating baselines
When a replay fails, compare its trace to the last passing run: changed input interpretation, selected branch, retrieved evidence, tool payload, state mutation, or response policy. Classify the result as intended product change, test defect, environment drift, or regression. Only update the expected fixture after a reviewed decision explains why the prior promise changed. Otherwise a ‘bless’ operation can convert a safety regression into a passing build. Rasa’s test-case conversion documentation presents conversion as a beta feature; generated cases still need human review for assertions, sensitive content, and representative coverage.
Run a small, meaningful suite
Tag fixtures by risk and run the smallest relevant set on each change, with a wider scheduled suite for integrations. Keep a separate set of preserved hard cases that reviewers inspect when changing prompts, retrieval, or policy. Report fixture count, pass/fail status, skipped environments, and known nondeterministic allowances, but do not turn coverage into a quality claim by itself. The editorial recommendation is to let real failures create test assets while maintaining consent, minimization, and review boundaries. Official product testing documentation explains available mechanics; your acceptance criteria must remain tied to the actual user and action risk.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.