让大模型真正理解图文演示中的视觉信息,突破文本依赖瓶颈
True Multimodal In-Context Learning Needs Attention to the Visual Context
- 通过动态重分配注意力,强制模型关注图像内容而非仅模仿文字
- 在新构建的TrueMICL数据集上,多模态理解能力提升37.2个百分点
- 适合研究多模态学习、评估真实跨模态适应能力的学者使用
多模态大语言模型(MLLMs)虽具备从少量图文示范中学习新任务的能力(即多模态上下文学习,MICL),但当前模型常忽视图像信息,过度依赖文本模式,导致学习行为仍为单模态。这种现象在标准视觉-语言数据集上被性能提升所掩盖,而实际需融合视觉信息的任务中表现不佳。为此,本文提出动态注意力重分配(DARA)策略,通过调整视觉与文本标记间的注意力分布,促进对视觉上下文的关注。同时构建了专门用于评估真正的多模态上下文学习的TrueMICL数据集,其支持集和测试集均要求整合图像内容才能正确作答。大量实验表明,该方案显著提升了模型的真实多模态适应能力,在TrueMICL上达到68.4%准确率,相比基线提升37.2个百分点。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demonstrations consisting of images, questions, and answers. Despite showing noticeable improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations. Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation. This behavior makes MICL still unimodal and largely restricts its practical utility. More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual context. As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored. To address these issues, we first introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens. In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal information-particularly visual content-for correct task completion. Extensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the true multimodal in-context learning capabilities. Code and datasets are available at https://chenxshuo.github.io/true-micl-colm .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。