Evaluation & operations · Explore this field ↗ · Operations · 2 min read
Latency, resilience, and cost are one design problem
Every extra model, retrieval call, voice step, and tool adds time, failure probability, and spend.
Budget the turn
Break latency into capture, routing, retrieval, generation, tool execution, and rendering. Set targets and timeouts for each. A fast first token does not compensate for a slow final action, but a truthful progress signal can reduce abandonment. Voice and animated characters add speech recognition, synthesis, and motion timing; they can multiply both delay and variable cost.
Degrade by capability
When retrieval is unavailable, do not answer policy questions from memory. When a write tool is down, preserve a draft and offer a ticket. When speech fails, keep text. When animation drops frames, conversation should continue. Each dependency needs a safe degraded state, not one universal apology.
Cost follows conversation shape
Long context, repeated retrieval, verbose output, realtime audio, and rendered presence all create variable cost. Track cost per completed task and per failed task, not only per token. Cache stable public answers with freshness controls. Summarize old turns, but preserve transaction state separately so compression cannot alter commitments.
Operator note
Load-test the full path with realistic turn length and concurrency. Inject timeouts, partial failures, stale cache, and duplicate messages. Publish service limits to support staff so humans know whether to wait, retry, or take over.
A graceful degradation matrix
For each dependency, write the normal behavior, timeout, fallback, user message, operator alert, and recovery. If search fails, the agent may offer navigation but must not invent policy. If account lookup fails, it can save a draft question without pretending to open a case. If synthesis fails, text can remain available. If a write returns unknown, reconciliation takes priority over retry. Run this matrix during load tests, because cascading retries can turn a partial outage into both a service failure and a cost spike.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.