arXiv:2606.11386cs.CLcs.AI2026-06被引 1

发现语音模型响应延迟根源,用简单方法显著提升打断理解能力

Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

论文配图:Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
图 1 · 摘自论文原文
  • 通过分析隐藏状态预测模式,发现模型在听和说时有不同注意力倾向
  • 提出感知向量干预,使模型在用户打断时响应正确率从28%升至45%
  • 无需微调即可生效,适合追求实时交互体验的语音系统研发者

全双工语音语言模型(FD-SLM)实现语音交互的无缝切换,但其内部听与说的协调机制尚不明确。我们分析了隐藏表示中的预测行为,发现其具有流特异性:听时优先预测用户输入流,说时则优先预测模型输出流。基于此,我们揭示模型动态调节生成态与感知态,但对对话突变响应滞后——用户打断时,模型仍短暂偏向生成态,导致错过输入起始部分,称之为状态惯性。为量化影响,我们提出零缓冲基准(ZBB),评估突发打断下的理解能力,以响应正确率和首词出现率(IWOR)衡量。最后,我们采用感知向量进行激活引导,一种无需训练的轻量级干预。在多个先进FD-SLM上,该方法显著改善打断处理:例如在PersonaPlex上,正确率从28%提升至45%,首词出现率从40%升至72%,且无需微调。

原文摘要 · Abstract (English)

Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coordinate listening and speaking remains underexplored. We analyze the predictive behavior encoded in FD-SLM hidden representations and find that they exhibit stream-specific predictive patterns: during listening, they preferentially predict the incoming user stream, whereas during speaking, they preferentially predict the model output stream. Building on this observation, we show that FD-SLMs dynamically modulate their internal predictive focus between two states: a generative state aligned with model output generation and a perceptive state aligned with incoming user input. However, this modulation can lag behind abrupt changes in conversational context. During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it to miss the beginning of the incoming input. We term this delayed internal transition state inertia. To quantify its downstream impact, we introduce the Zero-Buffer Benchmark (ZBB), a diagnostic benchmark for evaluating immediate interruption comprehension when user speech begins abruptly. We evaluate this setting using response correctness and initial-word occurrence rate (IWOR). Finally, we mitigate state inertia through activation steering with a perception vector, a training-free intervention with little additional computational overhead. Across multiple state-of-the-art FD-SLMs, activation steering substantially improves interruption handling; for example, on PersonaPlex, it improves correctness from 28% to 45% and IWOR from 40% to 72% without any fine-tuning.

语音模型实时交互状态惯性激活引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。