用信息分解法揭示多模态模型依赖视觉的深层原因
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
- 通过条件化语言的感官信息分解，分离视频与音频的独有、冗余和协同贡献
- 发现决策仍主要依赖模态独有信息，视觉影响远强于音频，存在感官协同瓶颈
- 适合研究多模态模型可解释性、融合机制及提升跨模态推理的研究者
理解多模态大语言模型如何利用不同模态对可靠推理至关重要。本文采用部分信息分解（PID）作为决策层分析工具，提出感官信息分解（Sensory PID），该方法以语言为条件，分离视频与音频的独有、冗余和协同贡献。在全模态模型上的应用显示：即使在视听融合任务中，决策仍主要由模态独有信息主导，且对视觉的依赖更强。模态打乱干预进一步验证了这种不对称性；分层分析揭示视觉优先的计算模式；指令扰动实验表明，晚期感官融合受语言条件制约。除诊断外，基于PID的样本重加权初步证明局部诊断信号可提升多模态推理与定位性能。作为参考验证，视觉-语言分析在多个任务、模型、干预和层级上广泛复现了此前报告的决策层PID模式。
原文摘要 · Abstract (English)
Understanding how multimodal large language models use different modalities is important for reliable reasoning. We employ Partial Information Decomposition (PID) as a decision-level lens and introduce Sensory PID, a conditional formulation that conditions on language and separates unique, redundant, and synergistic contributions from video and audio. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck: even on audio-visual fusion tasks, decisions remain dominated by modality-unique information, with stronger reliance on vision. Modality-shuffling interventions support this asymmetry, while layer-wise analysis reveals a visual-first computation pattern and instruction perturbations show that late-stage sensory fusion is conditioned by language. Beyond diagnosis, PID-guided sample reweighting provides initial evidence that local diagnostic signals can improve multimodal reasoning and grounding performance. As reference validation, our vision-language analysis broadly corroborates previously reported decision-level PID patterns across tasks, models, interventions, and layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。