Evaluation & operations · Explore this field ↗ · Operations · 2 min read
Observe the service without building a surveillance machine
Log enough to diagnose behavior, then minimize, protect, sample, and expire.
Prefer events over indiscriminate text
A useful operational record can include session pseudonym, turn timing, route, retrieval IDs, tool name, policy decision, handoff reason, and outcome without duplicating every message. Store raw text only when a defined review purpose requires it. Redact credentials and payment data before logs leave the request boundary.
Make access inspectable
Separate customer-service access from model-debugging access. Use role controls, short-lived links, audit trails, and sampled review queues. Audio deserves special care because voice can identify a person even when the transcript is redacted. Amazon Lex documents separate text and audio logs, encryption options, and a COPPA restriction—details that illustrate why logging configuration is a governance decision.
Monitor the failure distribution
Aggregate metrics should retain channel, language, task, product, and accessibility dimensions where lawful and appropriate. A falling average handoff rate can hide worsening outcomes for complex cases. Review false containment: sessions marked resolved because the user left, not because the issue was solved.
Operator note
Publish a log dictionary internally. For each field: purpose, source, sensitivity, viewers, retention, deletion, and alert use. Rehearse an incident involving transcript exposure and verify that derived stores can be found.
Sample with a purpose
Create review queues for defined questions: unsupported answers, failed tools, repeated clarification, handoff quality, and policy exceptions. Sample enough successful sessions to detect false positives, then expire raw text after adjudication unless a case requires retention. Reviewers should see only fields needed for that queue. Their labels feed the evaluation set and source backlog, not an unbounded employee search tool. Publish access reports internally so operational curiosity does not quietly become permanent transcript surveillance.
Separate debugging from performance management
Conversation evidence collected to diagnose the system should not silently become a tool for ranking individual employees or customers. Secondary uses need their own purpose, access decision, and notice. Otherwise the safest reviewer behavior becomes avoiding the record that the team needs to improve.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.