Conversation design · Explore this field ↗ · Implementation · 3 min read
Authoring interruptible voice turns: VAD versus semantic endpointing
Choose turn boundaries by task and failure cost, then make barge-in, partial speech, tools, and cancellation obey the same state machine.
See the source / a related case
A silent gap is not necessarily an endpoint
Voice activity detection (VAD) answers an acoustic question: is there speech-like sound now? It is excellent for beginning to listen and for noticing that a user has started talking over the agent. It cannot by itself know whether a silence ends a thought, follows a difficult word, reflects background noise, or precedes a correction. Semantic endpointing adds evidence from recognized content and conversational context to predict that a turn is complete. LiveKit’s current turn documentation explicitly separates VAD-only, speech-to-text endpointing, a turn detector model and manual control. Treat these as authoring choices, not a contest with one universally natural setting.
Choose by task consequence
Use VAD-only behavior when low latency or broad language coverage matters and an early reply is cheap to repair. Consider semantic or speech-to-text endpointing for open-ended explanations, corrections and multi-clause requests where a fixed silence window creates costly cutoffs. Use manual push-to-talk or a visible send control when a user must dictate an identifier, formulate an assistive communication message, work in a noisy environment, or control a consequential action. A technical endpoint should never itself authorize a purchase, deletion, booking or consent. Those flows need a readable review and an explicit confirmation after the speech has been interpreted.
Barge-in has a separate meaning
When the agent is speaking, VAD may notice an incoming sound immediately, but that sound may be a backchannel such as ‘mm-hm’ rather than a request to yield. LiveKit documents an adaptive mode intended to distinguish intentional interruptions from brief acknowledgements after VAD detects incoming audio. That behavior can improve a particular deployment, but it remains a decision policy with edge cases. Preserve a deterministic escape hatch: stop button, keyboard control and a way to say or type ‘wait’ or ‘repeat’. In an embodied interface, stop queued mouth motion and gestures at the same time as playback so the character visibly yields.
Interrupting speech does not necessarily cancel work
State what happens when a user speaks during a tool call. LiveKit’s tool guidance notes that an interrupted agent can leave a tool running in the background unless application logic cancels it; that is safe only for operations whose continued execution is understood and communicated. Categorize tools as cancellable, non-cancellable but reversible, and non-cancellable irreversible. For the latter, block interruption only around the smallest atomic commit, show a clear pending status, use idempotency, and reconcile before retrying. Never imply ‘cancelled’ merely because the audio stopped.
Measure repair and visible state
Log endpoint mode, timing, recognition finalization, barge-in detection, agent-audio stop, tool status and final task outcome. Sample sessions where people restart, speak during playback, or abandon after a pause. Tune one cohort and task at a time; a threshold that works for a casual concierge may be unacceptable for dictated addresses. Vendor behavior and model support evolve, so verify the current implementation documentation before treating a configuration as production guidance.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.