发现并解决多模态模型偏信文字、忽视图像的问题
Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
- 从训练方法角度分析视觉与文本推理失衡机制
- 改进数据与损失设计可显著缩小模态差距
- 适合关注多模态模型公平性与推理能力的研究者
多模态大语言模型（MLLMs）在视觉-语言任务中表现出强大能力，但近期研究发现其在视觉与文本模态的推理能力存在失衡。当前模型往往过度依赖文本线索而忽略视觉内容，导致需要真实视觉推理的任务表现不佳。本文将此现象称为‘模态差距’，即以文本为中心和以视觉为中心输入间的性能差异。通过训练配方视角分析，我们发现现有训练策略会加剧这一差距，并从数据与损失设计两个互补方向系统探索了弥合方法。研究结果为构建更均衡的多模态推理训练方案提供了新思路。代码已公开于 https://github.com/UCSB-NLP-Chang/Bridging-Modality-Gap。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have demonstrated strong capabilities on vision-and-language tasks. However, recent findings reveal an imbalance in their reasoning capabilities across visual and textual modalities. Specifically, current MLLMs often over-rely on textual cues while under-attending to visual content, resulting in suboptimal performance on tasks that require genuine visual reasoning. We refer to this phenomenon as the \textit{modality gap}, defined as the performance disparity between text-centric and vision-centric inputs. In this paper, we analyze the modality gap through the lens of training recipes. We first show that existing training recipes tend to amplify this gap. Then, we systematically explore strategies to bridge it from two complementary perspectives: data and loss design. Our findings provide insights into developing training recipes that mitigate the modality gap and promote more balanced multimodal reasoning. Our code is publicly available at https://github.com/UCSB-NLP-Chang/Bridging-Modality-Gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。