Teams that build a chat-based dialogue system and then add a voice channel by routing ASR (automatic speech recognition) output through the same NLU pipeline consistently underestimate the rework required. A model that handles chat input at 94% accuracy may drop to 79% on voice input from the same users in the same domain. The failure modes are different, the error distributions are different, and the dialogue management adaptations required are different. This article is about why, and what to do about it.
The ASR Error Problem
Text from a chat channel is exactly what the user typed. Voice input passes through ASR first. ASR introduces word error rates typically between 3-12% depending on audio quality, speaker accent, background noise, and whether the model is domain-adapted. At 5% WER on a 15-word utterance, there is roughly a 54% chance that at least one word in the utterance is transcribed incorrectly. That error might be in a filler word that does not affect intent, or it might be in a slot value that changes the meaning entirely.
The NLU model trained on clean text does not know that the input it receives from a voice channel may contain substitution errors. A named entity extractor trained on typed text may fail when "San Francisco" is transcribed as "San Fransisco" or "sent Francisco." A date extractor may fail when "Thursday the 12th" is transcribed as "Thursday the 12." Intent confidence scores are not calibrated to account for ASR errors, so the model may classify confidently on a subtly wrong transcript.
The standard mitigation is ASR confusables training: during NLU model training, augment the training examples with ASR-confusable variants of key entities and phrases. If your domain involves flight booking, add training examples where "LAX" is variously transcribed as "lax," "L.A.X.," "el ay ex," and phonetic variants. This makes the model more robust to ASR noise without requiring any changes to the ASR component itself.
Turn-Taking Signals Are Different
In chat, turn boundaries are explicit: when the user submits a message, the turn ends. In voice, turn boundaries are detected by end-of-speech detection algorithms, which use silence duration and prosodic cues to decide when the user has finished speaking. End-of-speech detection errors produce two failure modes: premature cutoffs (the system interrupts the user mid-sentence) and excessive latency (the system waits too long after the user finishes, producing an awkward pause).
Premature cutoff is the more damaging failure. When the user says "I'd like to book a flight from" and the system interrupts to ask for a destination, the user's mental model of the conversation breaks down. The system appeared to understand the utterance before it was finished. In chat, this failure mode cannot occur. In voice, it occurs regularly when end-of-speech thresholds are configured too aggressively.
The NLU implication: voice turns frequently arrive as partial utterances, either because of cutoffs or because users naturally pause mid-sentence while thinking. The intent classifier must handle partial utterances gracefully: identify the most probable intent given incomplete information, withhold slot extraction until the utterance is complete, and prompt for completion if the turn appears to be partial. None of this is needed for chat input where partial turns simply do not occur.
Disfluencies: The Vocabulary That Does Not Appear in Chat Training Data
Spoken language contains disfluencies: filler words ("um," "uh"), repetitions ("I want to, I want to book"), self-corrections ("book a flight to - actually, train to Chicago"), and restarts ("cancel - wait no, I mean change my reservation"). Chat users occasionally produce similar patterns but at dramatically lower rates. Voice users produce disfluencies constantly - they are a normal feature of spoken language production.
An NLU model trained on clean text will misparse disfluent input in two ways: entity extractors may extract partial or repeated entities, and intent classifiers may be distracted by filler words that change the probability distribution over intents. "I um want to cancel uh my flight" should produce the same classification as "I want to cancel my flight." Training on disfluency-free text does not guarantee this.
Disfluency normalization - a preprocessing step that strips or normalizes filler words, repetitions, and restarts before NLU processing - is standard in production voice NLU pipelines. The normalization rules are language-specific and need to be validated on real voice transcripts from your domain, not just generic spoken language corpora.
Prosodic Context and Dialogue State
Voice has a channel of information that chat does not: prosody. Stress, intonation, and speaking rate carry meaning that is absent from transcribed text. "I want to CANCEL my flight" (emphasized "cancel") expresses stronger intent than "I want to cancel my flight." "The reservation is on TUESDAY?" (rising intonation) expresses confirmation-seeking, not a statement. "The 12th" spoken with hesitation suggests lower certainty than the same phrase spoken at normal pace.
Current production voice NLU systems rarely incorporate prosodic features into intent classification, primarily because ASR output is typically text, and the acoustic features that carry prosodic information are discarded at the transcription stage. The field is moving toward end-to-end audio models that process the audio directly without a text transcription step, preserving prosodic information. Until such models reach production-grade reliability for task-oriented dialogue, prosodic context is effectively lost at the ASR boundary.
Latency Requirements Are Tighter for Voice
Chat users tolerate 500-800ms response latency without noticeable friction. Voice users perceive delays above 300ms as awkward pauses. Voice above 500ms feels like the system did not hear them. The entire NLU + context management + response generation pipeline must fit within 250ms on the critical path, leaving 50ms for TTS (text-to-speech) synthesis. This is half the latency budget available for equivalent chat deployments.
As we discussed in our article on keeping context state updates under 10ms, the context layer must be fast enough to leave adequate headroom for model inference. For voice, the constraint is even tighter. Context operations above 20ms start degrading the available inference budget to the point where only sub-3B models are feasible, which materially affects response quality.
What Adapting a Chat NLU System for Voice Actually Requires
Based on our work adapting the Equmenopolis context management layer for voice channel deployments, the adaptation requires: ASR confusables augmentation in NLU training data (2-3 weeks), disfluency normalization preprocessing (1 week), end-of-speech threshold tuning for your specific deployment environment (1-2 weeks of production tuning), partial utterance handling in intent classification (1 week), and latency profiling and optimization to hit voice latency targets (2-3 weeks). Total: 7-10 weeks of focused effort for a team already familiar with the chat deployment.
This estimate assumes the core NLU model quality on clean text is already acceptable. If the chat model has accuracy problems, those must be resolved before voice adaptation - voice adds noise on top of an already-degraded signal, not on top of a clean one.
Conclusion
Voice and chat are not the same channel with different input methods. They have different noise profiles, different turn-taking semantics, different vocabulary distributions, and different latency requirements. Treating voice as "chat with ASR in front" is the reason voice deployments consistently underperform relative to expectations. The adaptations are tractable - they are mostly known engineering problems - but they require deliberate effort, not optimism. Build the chat system first. Build the voice adaptation as a separate, planned workstream. Plan for 8 weeks, not 2.