Evaluation & operations · Explore this field ↗ · Operations · 2 min read
Metrics that do not lie about resolution
Containment is not success when the customer gives up, repeats the task, or fixes the problem elsewhere.
Define an outcome from the user’s job
For information tasks, verify that the answer was supported and useful. For actions, read status from the system of record. For triage, measure correct routing and context transfer. User confirmation helps but can be biased by fatigue or interface pressure. No single signal proves resolution.
Track cost shifted to people
A bot can lower queue volume while making remaining cases harder. Measure time humans spend reconstructing context, correcting misinformation, and reversing actions. Include customer effort, turns to resolution, repeated questions, repeat contact within a task-specific window, and escalation wait.
Segment before celebrating
Review performance by task, channel, language, account type, accessibility route, and consequence. A high-volume password FAQ can hide poor billing or cancellation behavior. Google’s experiment metrics include no-match, handoff, callback, abandonment, and session end; use such operational signals as a starting set, not as proof of customer success.
Operator note
Write a metric contract stating numerator, denominator, exclusions, attribution window, and data source. Sample transcripts behind every dashboard movement. Pair an efficiency measure with a user outcome and a risk measure.
Read a metric as a story
Suppose containment rises while repeat contact and cancellation complaints also rise. The useful question is not whether the bot handled more sessions; it is which tasks moved, why users returned, and whether a route blocked access to people. Sample the new contained cohort, compare downstream account state, and inspect language and channel segments. Metrics should trigger investigation, not write the conclusion. A dashboard earns trust when operators can move from a percentage to representative conversations and system records without exposing more personal data than necessary.
Report uncertainty
Small cohorts, delayed outcomes, missing feedback, and changing traffic mix limit interpretation. Put sample size and observation window beside the percentage. When a vendor defines a billable resolution, keep that commercial definition separate from the organization’s customer-outcome definition.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.