Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations
The paper introduces HIVE (Human Input-Variation Engine), a suite of voice transcription perturbations and QWERTY keyboard perturbations. It evaluates LLM robustness and reports seven findings: (i) voice transcription perturbations lower accuracy across all instruction-tuned models tested, with transcription structure being the main cost driver; (ii) QWERTY keyboard perturbations cost less, with models absorbing many before accuracy drops; (iii) both channels' impact is tied to how many question tokens survive the perturbation—destroying a token hurts, adding new ones costs little; (iv) the gap between channels appears only in constructive or deductive answer tasks, not multiple choice; (v) the harm is not solely from test-set contamination.
A study compares how voice and keyboard input perturbations affect LLM performance. Voice transcription perturbations consistently reduce accuracy across instruction-tuned models, primarily due to structural changes rather than fillers. Keyboard perturbations are less damaging, with models tolerating many errors before accuracy declines. The key factor is token survival: destroying tokens harms performance, while adding tokens has minimal impact. The performance gap between input channels emerges only in tasks requiring constructed or deduced answers, not in multiple-choice settings. The observed harm is not fully explained by test-set contamination.
The study isolates token survival as the primary mechanism behind input perturbation impact. Voice transcription perturbations alter the structure of the input, leading to token destruction, which directly degrades model accuracy. Keyboard perturbations, even when numerous, preserve more original tokens and thus cause less harm. The finding that the input channel gap disappears in multiple-choice tasks suggests that the perturbation effect is task-dependent, likely because multiple-choice questions rely less on precise token-level reasoning.
For applications relying on voice input, such as voice assistants or dictation-based interfaces, the robustness of LLMs to transcription errors is a critical concern. The results imply that current instruction-tuned models are vulnerable to realistic voice input variations, which could affect user experience and reliability. Developers may need to invest in better transcription systems or model fine-tuning to mitigate these effects. The relative resilience to keyboard typos suggests that text-based interfaces may currently offer more consistent performance.
The findings highlight a potential barrier to voice-based AI products: if models perform worse with voice input, user satisfaction and task completion rates may suffer. Companies developing voice assistants, customer service bots, or dictation tools may need to prioritize input robustness to maintain competitive performance. Conversely, text-based interfaces may have a reliability advantage, influencing product design decisions.
Future work may focus on improving model robustness to structural perturbations in voice input, possibly through training on perturbed data or architectural changes. The HIVE suite could become a standard benchmark for evaluating input robustness. As voice interfaces become more prevalent, addressing this vulnerability will be essential for deploying reliable LLM agents in real-world settings. Observing whether model updates reduce the voice-keyboard gap will be a key signal of progress.