让视觉语言模型学会像人一样推理:从例子中找规律再应用。
Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning

- 用相似性压缩视觉片段,剔除冗余信息
- 动态重分配注意力,让每张图都得到公平关注
- 通过思维链引导模型先归纳规则再解题,提升可靠性
上下文学习(ICL)使大模型能用少量示例适应新任务,但其在视觉语言模型(VLMs)中的应用仍不稳定。分析发现根本瓶颈在于归纳鸿沟:模型常基于错误推理得出正确答案,却难以从示范中提取一致规律。这一问题在视觉层面更突出:冗余视觉标记过多遮蔽文本线索,注意力分布又过度偏向首张图像。为此,我们提出一个重构多模态ICL的归纳-演绎框架。该框架包含基于相似性的视觉标记压缩模块,用于过滤冗余图像块;动态注意力重平衡机制,确保所有图像获得均衡关注;以及链式思维范式,引导模型逐例分析、推导通用规则,再应用于查询。此外,辅以监督微调与可验证奖励的强化学习管道,强化准确引用和噪声过滤。在涵盖视觉感知、逻辑推理、STEM问题及反讽识别的八个基准上测试,对多个开源VLM均实现显著且一致的性能提升,表明赋予模型真正的归纳能力在多模态场景中的潜力。
原文摘要 · Abstract (English)
In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile. Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning, while struggling to extract consistent rules across demonstrations. This gap is further exacerbated by two visual-level obstacles: an overwhelming proportion of redundant visual tokens that obscure textual cues, and a skewed attention distribution that favors the initial image at the expense of subsequent context. To address these issues, we introduce a framework that restructures multimodal ICL as a principled inductive-deductive process. The framework incorporates a similarity-based visual token compression module to filter out redundant patches, a dynamic attention rebalancing mechanism to distribute focus equitably across all images, and a chain-of-thought paradigm that explicitly guides the model to analyze individual examples, derive a generalizable rule, and then apply it to the query. An auxiliary learning pipeline combines supervised fine-tuning with reinforcement learning using verifiable rewards to reinforce faithful citation and noise filtering. Evaluations across eight benchmarks covering visual perception, logical reasoning, STEM problems, and sarcasm detection demonstrate consistent and significant improvements over standard ICL baselines for multiple open-source VLMs, highlighting the potential of equipping models with genuine inductive capabilities in multimodal settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。