音频视频冲突下大模型易受内部先验干扰，导致判断失误。
Compositional Failure in Audio-Visual LLMs: Late-Layer Prior Dominance Under Cross-modal Conflict
- 通过跨模态冲突测试，发现模型依赖内部先验而非真实输入。
- 在视频语音对齐任务中，准确率下降32.3%，指令遵循失败率达17.3%。
- 问题源于深层网络过早固化答案偏好，适合关注多模态鲁棒性的研究者。
本文将音视频冲突作为音视频大模型的组合泛化测试：模型需结合同步但语义冲突的音视频证据，判断是否匹配。在VideoLLaMA 2-7B-AV上，三种对齐配置在AVHBench的精确字符串评分子集上仍接近随机水平，尽管其输出先验发生显著变化。类似地，现成的InternVideo2在跨模态冲突下准确率下降32.3%，指令遵循失败率上升17.3%。我们将其归因于先验主导现象：深层网络在缺乏充分输入支撑的情况下，固守内部偏好答案模式。通过机制可解释性分析发现，这种承诺集中于25.5 ± 1层。更强的时间对齐虽能改变答案偏差，但无法提升组合冲突解决能力。代码与数据可在https://github.com/AdarshSudheer09/AVHBench-dmai获取。
原文摘要 · Abstract (English)
We study audio-visual conflict as a compositional generalization test for AV-LLMs: the model must combine synchronized but semantically incompatible audio and video evidence and decide whether the pair matches. On VideoLLaMA 2-7B-AV, three alignment configurations remain nearchance on the scored exact-string Yes/No subset of AVHBench, even though their output priors shift substantially. Similarly, off-the-shelf InternVideo2 experienced a 32.3% accuracy decrease specifically under cross-modal conflict, accompanied by a 17.3% instruction-following failure. We call this failure mode prior dominance: late-layer commitment to an internally preferred answer pattern that is weakly grounded in the conflicting inputs. To explain this behavior, we conduct a mechanistic interpretability analysis and find that commitment remains concentrated at 25.5 $\pm$ 1 layers. We show that stronger temporal alignment changes answer bias, but do not improve compositional conflict resolution. Code and data to reproduce our mechanistic audit and behavioral evaluations are available at https://github.com/AdarshSudheer09/AVHBench-dmai.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。