让AI看图推理时不跑偏,靠动态调用视觉信息提升准确性
Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework
- 用大模型当思维指挥官,按需向视觉观察员要细节,不依赖早期文本
- 数学图像任务上准确率达47.2%,超过GPT-4o的40.6%
- 无需训练或改结构,插件式使用,适合想提升多模态推理的开发者
随着大语言模型(LLMs)的发展和强化学习的进步,其推理能力显著增强,可解决复杂问题。受此启发,研究者将相关技术拓展至大视觉语言模型(LMMs)。然而,一个关键缺陷浮现:视觉语义逐渐退化。随着推理链变长,模型越来越依赖早期生成的文本信息,而初始提取的视觉内容很少被重新利用或整合,导致推理偏离真实图像内容,产生不合理甚至错误结论。为克服这一根本局限,本文提出一种无需训练的代理式解耦框架(DRP),将认知推理与视觉感知分离。在此框架中,强大的大语言模型作为策略推理器,通过主动调用一个充当专职观察员的LMM来按需获取细粒度视觉细节。该方法轻量、模型无关且即插即用,无需额外训练或架构修改。大量实验表明,该框架能有效调控视觉推理轨迹,显著缓解推理漂移,强化视觉锚定。在MathVision基准上,Qwen2.5-VL-7B与Qwen3-32B的集成达到47.2%准确率,优于GPT-4o的40.6%。结果证明该方法可在不重训的前提下显著提升多模态推理可靠性。代码已开源:https://github.com/hongruijia/DRP。
原文摘要 · Abstract (English)
With the continuous expansion of Large Language Models (LLMs) and advances in reinforcement learning, LLMs have demonstrated exceptional reasoning capabilities, enabling them to address a wide range of complex problems. Inspired by these achievements, researchers have extended related techniques to Large Multimodal Models (LMMs). However, a critical limitation has emerged, reflected in the progressive loss of visual grounding. As the reasoning chain grows longer, LMMs tend to rely increasingly on the textual information generated in earlier steps, while the initially extracted visual information is rarely revisited or incorporated. This phenomenon often causes the reasoning process to drift away from the actual image content, resulting in visually implausible or even erroneous conclusions. To overcome this fundamental limitation, we propose a novel, training-free agentic paradigm that Decouples cognitive Reasoning from visual Perception (DRP). In this framework, a powerful LLM serves as a strategic Reasoner, orchestrating the inference process by explicitly querying an LMM-acting as a dedicated Observer-to retrieve fine-grained visual details on demand. This approach is lightweight, model-agnostic, and plug-and-play, necessitating no additional training or architectural modifications. Extensive experiments demonstrate our framework DRP's efficacy in regulating the visual reasoning trajectory, significantly mitigating reasoning drift, and enforcing robust visual grounding. Notably, on the MathVision benchmark, the integration of Qwen2.5-VL-7B and Qwen3-32B achieves an accuracy of 47.2\%, outperforming GPT-4o's 40.6\%. These findings underscore the potential of our approach to enhance multimodal reasoning reliability without the need for costly retraining. Our code is publicly available at https://github.com/hongruijia/DRP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。