arXiv:2604.16264cs.CVcs.LG2026-04

通过路由互补信息解决视觉语言模型的模态失衡问题。

Information Router for Mitigating Modality Dominance in Vision-Language Models

论文配图:Information Router for Mitigating Modality Dominance in Vision-Language Models
图 1 · 摘自论文原文
  • 设计信息路由机制,主动补充弱模态信息。
  • 在多个基准上提升模态平衡性,降质情况下仍保持性能。
  • 适合需要鲁棒多模态推理的应用场景。

视觉语言模型在多个基准测试中表现优异,但常因模态主导问题导致预测过度依赖单一模态。现有方法主要通过调整注意力分配来缓解,但仅改变关注位置,无法解决信息缺失或模糊的问题。真实场景中,各模态的信息密度和信噪比差异显著,单纯调整注意力无法根治信息不足。本文提出多模态信息路由器(MoIR),一种在融合前显式减少模态间信息差异的方法。MoIR识别信息贫乏的模态项,并从强模态中路由互补信息,构建信息丰富的表示,再交由大语言模型处理。通过调节信息可用性,即使某一模态退化,也能实现可靠的模态主导转移。我们在三个常用多模态基准上评估了MoIR,涵盖多种模型主干。实验结果表明,MoIR能持续实现更均衡的模态贡献,提升鲁棒性与下游性能,尤其在模态退化时优势明显。这些发现证明,显式修改跨模态信息是缓解多模态推理模型中模态主导的有效且互补策略。

原文摘要 · Abstract (English)

Vision Language models (VLMs) have demonstrated strong performance across a wide range of benchmarks, yet they often suffer from modality dominance, where predictions rely disproportionately on a single modality. Prior approaches primarily address this issue by steering model's attention allocation, implicitly assuming that all modalities provide sufficient information. However, attention only determines where the model focuses, and cannot enrich information that is missing or ambiguous. In the real world, input modalities often differ in information density and their signal-to-noise ratios. In such cases, simply adjusting model's attention does not resolve the underlying lack of information. In this paper, we propose \textsc{MoIR}: \textit{Multi-modal Information Router}, an information-level fusion method that explicitly reduces information disparity prior to fusion. \textsc{MoIR} identifies less informative tokens and routes complementary information from a stronger modality, constructing information-dense token representations before they are processed by a large language model. By modifying information availability, \textsc{MoIR} enables reliable shifts in modality dominance, even when one modality is degraded. We evaluate \textsc{MoIR} on three widely used multi-modal benchmarks across multiple model backbones. Experimental results show that \textsc{MoIR} consistently demonstrates more balanced modality contribution, and improves robustness and downstream performance, particularly even under modality degradation. These findings demonstrate that explicitly modifying cross-modal information is an effective and complementary strategy for mitigating modality dominance in multi-modal reasoning models.

多模态信息路由模态平衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。