发现语音大模型忽视声音信息，根源不在编码器，而在读取环节。
Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
- 更换编码器无法解决声音信息利用不足问题。
- 模型能恢复声音结构但生成结果仍差，最高差83分。
- 适合关注语音理解与模型可解释性的研究者。
语音条件语言模型常忽略语调、情感和非语音声音等声学线索，引发疑问：是否是基于ASR的前端在信息传入语言模型前就丢失了这些内容？我们通过在共享Qwen3.5-4B语音-语言模型管道中对比Whisper-Tiny、Whisper-Small与EnCodec、DAC-VAE、WavTokenizer，在语音识别、情感识别和声音描述任务上进行测试。仅替换编码器未能解决该问题，Whisper系列在整体表现上仍最优，包括情感与环境声音描述任务。为定位失败原因，我们追踪编码器、投影层、语言模型层及语言模型头部的任务相关信息。线性探测与几何分析表明，判别性声学结构在最终语言模型层仍可恢复，即使MCQA准确率比探测准确率低最多83分。由于答案格式与解码过程受控，这一任务依赖性差距指向内容特定的读取失败，而非通用格式偏差。LogitLens分析与针对性语言模型头部干预支持结论：声学信息利用不足并非仅由编码器侧信息丢失导致，读取对齐可能是主要瓶颈。
原文摘要 · Abstract (English)
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。