arXiv:2512.17227cs.CV2025-12被引 5

让AI学会何时看图,解决长链推理中视觉遗忘问题

Learning When to Look: A Disentangled Curriculum for Strategic Perception in Multimodal Reasoning

  • 分阶段训练:先学纯文本推理,再结合视觉线索构建思考路径
  • 用强化学习教模型在不确定时主动查看图像,提升视觉感知策略
  • 适合需要复杂多模态推理的场景,如医学影像分析、自动驾驶

多模态大模型在复杂长链视觉推理任务中表现脆弱,常出现‘视觉遗忘’现象——随着推理加深,模型逐渐丢失视觉依据,即‘想得越久,看得越少’。我们指出,这源于当前训练范式过早地将抽象逻辑推理(如何思考)与战略视觉感知(何时看图)混淆。为此,提出一种解耦式课程训练框架:首先在纯文本数据上建立稳健的抽象推理基础;随后引入感知锚定的思维链(PG-CoT)机制,将视觉信息与推理过程对齐。其次,将‘何时看图’建模为强化学习问题,设计关键感知奖励函数,通过语言标记(如‘等待’‘验证’)引导模型在认知不确定时主动观察图像,从而学习自主的视觉接地策略。该框架有效解决了模型冷启动缺陷与感知策略缺失,使模型从依赖直觉的观察者转变为具有战略性的可解释推理者。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) demonstrate significant potential but remain brittle in complex, long-chain visual reasoning tasks. A critical failure mode is "visual forgetting", where models progressively lose visual grounding as reasoning extends, a phenomenon aptly described as "think longer, see less". We posit this failure stems from current training paradigms prematurely entangling two distinct cognitive skills: (1) abstract logical reasoning "how-to-think") and (2) strategic visual perception ("when-to-look"). This creates a foundational cold-start deficiency -- weakening abstract reasoning -- and a strategic perception deficit, as models lack a policy for when to perceive. In this paper, we propose a novel curriculum-based framework to disentangle these skills. First, we introduce a disentangled Supervised Fine-Tuning (SFT) curriculum that builds a robust abstract reasoning backbone on text-only data before anchoring it to vision with a novel Perception-Grounded Chain-of-Thought (PG-CoT) paradigm. Second, we resolve the strategic perception deficit by formulating timing as a reinforcement learning problem. We design a Pivotal Perception Reward that teaches the model when to look by coupling perceptual actions to linguistic markers of cognitive uncertainty (e.g., "wait", "verify"), thereby learning an autonomous grounding policy. Our contributions include the formalization of these two deficiencies and the development of a principled, two-stage framework to address them, transforming the model from a heuristic-driven observer to a strategic, grounded reasoner. \textbf{Code}: \url{https://github.com/gaozilve-max/learning-when-to-look}.

多模态推理视觉感知强化学习思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。