arXiv:2608.03970cs.AI2026-08

比较语音和键盘输入对大模型影响,发现语音更易出错且难修复。

Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations

论文配图:Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations
图 1 · 摘自论文原文
  • 构建输入扰动工具HIVE,模拟真实语音与打字噪声。
  • 语音转写破坏关键词导致准确率下降,打字错误影响较小。
  • 模型需额外思考预算才能弥补语音损失,适合人机交互研究者。

人类通过打字或说话向语言模型输入信息,二者留下不同特征:键盘输入带来拼写错误;语音则因传统转录产生不连贯,以及由AI语音输入工具引发的句子重构。这些差异如何影响大模型性能?本文提出HIVE(人类输入-变异引擎),包含语音转写扰动与QWERTY键盘扰动,用于评估模型对此类扰动的鲁棒性。共得出七点发现:(i) 语音转写扰动降低所有指令微调模型的准确率,且影响源于转写结构而非填充词;(ii) 键盘扰动代价较低,模型可承受较多干扰而准确率仍稳定;(iii) 两种扰动的根源在于问题中多少关键词在扰动后保留——破坏关键词伤害大,添加新词影响小;(iv) 两通道差距仅出现在需推理或生成答案的任务中,选择题无显著差异;(v) 损失并非仅由测试集污染引起;(vi) 轻量级适配无法消除该影响;(vii) 增加思维预算可几乎完全恢复键盘通道表现,但对语音输入无效,压缩语音进一步加剧问题。

原文摘要 · Abstract (English)

Human input reaches language models by typing or speaking, and each channel leaves a distinct signature: orthographic noise for keyboards; for voice, disfluency from conventional transcription and restructuring from AI-backed dictation tools. How do they impact an LLM's performance? In this paper we present HIVE (Human Input-Variation Engine), a suite of voice transcription perturbations and QWERTY keyboard perturbations. We use HIVE to evaluate how robust models are to these perturbations. We present seven findings. (i) Voice transcription perturbations lower accuracy across every instruction-tuned model we test, and it is the structure of the transcription rather than its fillers that carries the cost. (ii) QWERTY keyboard perturbations cost less, and a model absorbs a lot of them before accuracy falls away. (iii) Both trace back to one cause, how many of the question's tokens survive the perturbation: destroying a token is what hurts, while adding new ones alongside it costs little. (iv) The gap between the two channels appears only where the answer must be constructed or deduced; on multiple choice there is none. (v) The harm does not solely come from test-set contamination. (vi) It cannot be trained away with lightweight adaptation. (vii) A thinking budget recovers the keyboard channel almost entirely but leaves the spoken registers untouched, and compressed speech is worse with it.

大模型语音输入输入扰动人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。