Audio LLM常误解语气情感，新基准揭示其依赖文字而非声音的缺陷
Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
- 构建2000例对抗性语音数据，精准测试模型对语调、情绪等非语言信息的理解
- 多数模型仅17%正确识别声音特征，严重依赖文本暗示的错误答案
- 提出分层融合与偏好优化方法，使模型更关注真实语音线索
音频大语言模型（Audio LLMs）在语音理解任务中表现优异，但对语调、情绪等副语言信息的理解能力仍有限。为系统量化该问题，我们提出VoxParadox，一个包含2000个经验证例的对抗性基准，覆盖10类副语言任务，通过可控语音合成制造文本与语调不一致的场景，实现对语音副语言理解能力的直接测量。评估多种Audio LLM发现其在声学真实答案上准确率普遍偏低，且强烈倾向选择语言暗示的错误答案。层间探测显示：(i) 副语言线索在深层编码器及编码器-语言模型接口处易衰减；(ii) 即便音频令牌中存在有效线索，语言模型也常忽略。为此，我们提出提示条件层混合器（PCLM），根据输入提示自适应融合多层音频信息，并结合直接偏好优化（DPO）显式偏好声学支持选项。该方法显著提升模型性能：使Audio Flamingo 3在VoxParadox上的准确率从17.40%提升至65.20%，在MMSU副语言子集上从37.74%提升至54.78%。
原文摘要 · Abstract (English)
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。