arXiv:2609.04247eess.AScs.SD2026-09

提出新解码框架CAD,减少多模态大模型的幻觉干扰。

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

论文配图:CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models
图 1 · 摘自论文原文
  • 通过冲突感知机制评估跨模态预测不一致程度
  • 在多个数据集上提升准确率,最高达14.1个百分点
  • 无需重训练,适合部署于现有多模态模型

多模态大语言模型(Omni-LLMs)融合音频、视频和文本,但仍易产生跨模态幻觉,即一种模态错误影响对另一模态的判断。现有无训练解码方法通过扰动或相关性加权调节模态影响,但未评估联合音视频分支内的预测兼容性。由于联合分支的差异可能源于有害干扰或有益互补,可靠干预需同时评估差异大小与可行动性。为此,本文提出无训练框架冲突感知解码(CAD),包含潜在冲突幅度估计(PCME)与冲突可行动性评估(CAA)。PCME利用音视频不一致及联合预测偏离加权单模态参考的程度量化潜在冲突;CAA则使用Dempster-Shafer可靠性折扣方法,结合查询相关性和答案确定性判断是否需要干预。当识别出可行动冲突时,CAD将解码权重从联合分支重新分配至单模态分支。在CMM、AVHBench、WorldSense和VideoMME上的实验表明,CAD在多个音视频骨干网络上均优于基线解码器和竞争性无训练方法。在Qwen2.5-Omni-7B上,CAD在CMM和AVHBench上分别提升整体准确率14.1和8.0个百分点,且无需模型重训练。

原文摘要 · Abstract (English)

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.

多模态幻觉抑制解码策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。