Make failure observable.

Evaluate conversations before launch

A demo proves possibility. A test set probes whether the service behaves correctly across ordinary and adversarial cases.

Observe the service without building a surveillance machine

Log enough to diagnose behavior, then minimize, protect, sample, and expire.

Latency, resilience, and cost are one design problem

Every extra model, retrieval call, voice step, and tool adds time, failure probability, and spend.

Metrics that do not lie about resolution

Containment is not success when the customer gives up, repeats the task, or fixes the problem elsewhere.

Rasa · source visual / reviewEvaluation & operations3 min

Conversation replay and regression testing: turn real failures into controlled fixtures

How to preserve the meaningful shape of a failed conversation, replay it safely, and assert behavior without pretending language systems are deterministic everywhere.

Async jobs: say queued, running, cancelled, or unknown

A practical contract for external work that outlasts one turn, with status language users can inspect and cancellation semantics engineers can keep honest.

Conversation quality review: use an anchored rubric, not a vibes score

A human review rubric can make task correctness, evidence, action safety, and repair quality discussable—and calibration can expose where the policy itself is vague.

Release gates and rollback: make stopping a change routine

A release package should say exactly what changed, what will be observed, who can stop it, and how to return to a known safe configuration without inventing a universal threshold.

Model-change evaluation: compare a changed system without a live leaderboard

When a model, configuration, or system prompt changes, hold the other inputs steady, compare paired cases, and inspect the slices that can be harmed by an average improvement.

Incident response for wrong actions: stop, verify, contain, reconcile

When automation may have changed the wrong thing, prioritize stopping further impact and verifying authoritative state before attempting any broad undo.

Multilingual evaluation fixtures: test the interface, not only translated sentences

A multilingual conversational system needs locale, direction, rendering, and review fixtures that make language-specific failures visible before release.

Latency trace budget: measure the first usable response

A timestamped trace separates observed critical-path delay from an illustrative serial budget before a team tries to optimize a conversational wait.

Webhook delivery reconciliation: separate receipt from business outcome

Authenticate and record a delivery once, then reconcile the business record that delivery describes instead of treating delivery mechanics as proof of state.

Retry and timeout budgets: preserve one deadline

Give each request a deadline, one retry owner, and a finite retry allowance so nested clients do not amplify a transient fault into a wider outage.

Retrieval abstention test set: make no-evidence measurable

Release grounded answers only after testing questions that should remain unanswered, conflicted, or unsupported. This field note supplies an inspectable artifact, counterexample, and release checks.

Synthetic conversation test data: useful fixtures without invented users

Create clearly fictional, nonidentifying fixtures while measuring what their construction fails to represent. This field note supplies an inspectable artifact, counterexample, and release checks.