通过对比音频与静音参照，抑制大模型在听觉问答中的幻觉回答。
TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models
- 用真实音频与静音对照，动态调整决策门控以减少误判。
- 在多个数据集上提升准确率，最高增益达0.117（F1）。
- 无需训练，适合追求可靠性的音频问答应用。
大型音频语言模型（LALMs）会幻觉生成不存在的音频对象，对缺失声音事件错误回答“是”，损害音频问答的可靠性。本文提出无需训练的消幻觉策略Token-Adaptive Decoding（TAD），通过对比真实音频与匹配静音参考的输出概率，使初始“是/否”判断更可信。TAD引入一种基于置信度的动态门控机制，在首个解码步骤起关键作用，且对肯定类词元具有类别条件性，利用音频-静音差异避免证据不足时的过度修正。在AudioCaps-Hallucination数据集上，相较固定对比强度的对比基线（AAD），TAD在Qwen2上使F1提升0.059至0.117，在Gemma上提升0.025至0.064；在Clotho-AQA上，Qwen2的F1从0.810升至0.816，Gemma性能与AAD持平。
原文摘要 · Abstract (English)
Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events, thus undermining reliability in audio question answering. We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation that grounds the initial yes/no decision by contrasting logits under real audio with a matched silent reference. TAD introduces a token-adaptive, confidence-guided gate that is decision-critical at the first decoding step and class-conditional on affirmative tokens, using the audio-silent margin to avoid overcorrection when evidence is weak or already sufficient. Experiments on AudioCaps-Hallucination show that, relative to Audio-Aware Decoding (AAD), a contrastive baseline with fixed contrast strength, TAD improves F1 for Qwen2 by 0.059 to 0.117 across Popular, Adversarial, and Random splits, and for Gemma by 0.025 to 0.064, while on Clotho-AQA it raises F1 from 0.810 to 0.816 on Qwen2 and remains comparable to AAD on Gemma.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。