用合成负样本训练模型，让语音大模型学会识别没听到的声音。
Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples
- 通过大语言模型生成合成负样本，增强音频理解模型对缺失声音的辨别能力。
- 在多个音频问答与推理基准上，幻觉率显著降低，性能保持优秀。
- 无需修改大模型参数，仅用轻量适配器，数据与计算成本更低。
近年来，语音感知大语言模型（ALLMs）能够处理和理解音频输入，但常会虚构不存在的声音事件，降低其在真实场景中的可靠性。为此，我们提出LISTEN（Learning to Identify Sounds Through Extended Negative Samples），一种类对比学习的训练方法，利用骨干大语言模型生成的合成数据，提升ALLMs区分存在与缺失声音的能力。与以往方法不同，该方法无需修改大语言模型参数，通过轻量级适配器高效融合音频表征。实验表明，LISTEN有效缓解了幻觉问题，同时在现有音频问答与推理基准上保持优异性能。此外，在数据和计算效率方面更具优势。
原文摘要 · Abstract (English)
Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs. However, these models often hallucinate non-existent sound events, reducing their reliability in real-world applications. To address this, we propose LISTEN (Learning to Identify Sounds Through Extended Negative Samples), a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds using synthesized data from the backbone LLM. Unlike prior approaches, our method requires no modification to LLM parameters and efficiently integrates audio representations via a lightweight adapter. Experiments show that LISTEN effectively mitigates hallucinations while maintaining impressive performance on existing audio question and reasoning benchmarks. At the same time, it is more efficient in both data and computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。