发现多模态推理的瓶颈在于融合不当,而非感知能力不足。
Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
- 按逻辑模式分类多模态推理,区分模态间信息分布与组合方式。
- 新增模态仅在提供独立充分路径时提升性能,冗余或链式推理反降效。
- 适合研究多模态模型内部机制、训练策略优化的学者参考。
多模态大语言模型(MLLMs)通过整合文本、视觉和音频等多元输入,有望增强推理能力。然而跨模态推理仍缺乏系统研究,现有结果矛盾:某些情况下增加模态提升性能,另一些则反而降低。这源于缺乏受控评估框架及对模型内部机制的分析,难以判断模态交互在何时何地促进或损害推理。本文提出一个基于逻辑的评估框架,将多模态推理划分为六类交互模式,分析事实如何在不同模态间分布并逻辑组合。实证表明,只有当额外模态提供独立且充分的推理路径时,性能才提升;而冗余或链式蕴含支持常导致性能下降。此外,推理以三种系统性方式退化:弱模态拖累整体表现、模态冲突引发偏好偏差、多模态联合信号无法有效融合。由此揭示两大核心缺陷:任务组合瓶颈(识别与推理无法单步完成)与融合瓶颈(早期融合引入偏见)。进一步实验发现,注意力机制未能编码事实有用性,但采用两步提示(先识别后推理)可恢复性能,验证了任务组合瓶颈;模态身份在早期层仍可恢复,软化早期注意力能改善推理,凸显融合偏见为另一失败模式。总体表明,集成而非感知是多模态推理的主要障碍,建议采用组合感知训练与早期融合控制作为改进方向。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio. Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance. These inconsistencies stem from a lack of controlled evaluation frameworks and analysis of models' internals to isolate when and why modality interactions support or undermine reasoning. We address this gap through a logic-grounded evaluation framework that categorizes multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined. Empirically, additional modalities enhance reasoning only when they provide independent and sufficient reasoning paths, while redundant or chained entailment support often hurts performance. Moreover, reasoning degrades in three systematic ways: weaker modalities drag down overall performance, conflicts bias preference toward certain modalities, and joint signals from different modalities fail to be integrated effectively. Therefore, we identify two core failures: task-composition bottleneck, where recognition and reasoning cannot be jointly executed in one pass, and fusion bottleneck, where early integration introduces bias. For further investigation, we find that attention patterns fail to encode fact usefulness, but a simple two-step prompting (recognize then reason) restores performance, confirming the task-composition bottleneck. Moreover, modality identity remains recoverable in early layers, and softening attention in early fusion improves reasoning, highlighting biased fusion as another failure mode. Overall, our findings show that integration, not perception, is the main barrier to multimodal reasoning, suggesting composition-aware training and early fusion control as promising directions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。