用音频感知解码减少大模型对音频内容的幻觉
Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding
- 通过对比有无音频时的生成概率，选择更依赖音频的词
- 在物体幻觉数据集上提升F1分数0.046至0.428
- 适合需要高可信音频问答的应用场景
大型音频-语言模型（LALMs）可同时接收音频和文本输入，并回答关于音频的问题。尽管现有LALMs在标准基准上表现良好，但存在严重幻觉问题，即模型会虚构音频中并不存在的内容。为缓解这一问题，本文提出音频感知解码（Audio-Aware Decoding, AAD），一种轻量级推理阶段策略。AAD通过对比有无音频上下文时的令牌预测概率，优先选择在音频存在时概率上升的词。我们在三个LALMs上对物体幻觉数据集进行实验，结果表明AAD使F1分数提升0.046至0.428。同时，在Clotho-AQA等通用音频问答数据集上，准确率提升5.4%至10.3%。我们还进行了详尽的消融研究，验证了各组件的有效性。
原文摘要 · Abstract (English)
Large Audio-Language Models (LALMs) can take audio and text as the inputs and answer questions about the audio. While prior LALMs have shown strong performance on standard benchmarks, there has been alarming evidence that LALMs can hallucinate what is presented in the audio. To mitigate the hallucination of LALMs, we introduce Audio-Aware Decoding (AAD), a lightweight inference-time strategy that uses contrastive decoding to compare the token prediction logits with and without the audio context. By contrastive decoding, AAD promotes the tokens whose probability increases when the audio is present. We conduct our experiment on object hallucination datasets with three LALMs and show that AAD improves the F1 score by 0.046 to 0.428. We also show that AAD can improve the accuracy on general audio QA datasets like Clotho-AQA by 5.4% to 10.3%. We conduct thorough ablation studies to understand the effectiveness of each component in AAD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。