提出视觉引导重思机制,让多模态模型更精准理解图像细节。
GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
- 引入视觉线索引导的迭代重思策略,动态修正推理过程。
- 在M$^3$CoT上达81.5%准确率,超越O4-mini模型。
- 适合需要多视角图像理解的通用多模态任务使用者。
尽管多模态大模型取得显著进展,主流模型在通用场景下的视觉主导型多模态推理任务中仍表现不佳,根源在于过度依赖逻辑与知识驱动的慢思考策略,难以有效整合视觉信息。为此,本文提出GThinker(General Thinker),一种在通用场景、数学和科学领域均表现出色的多模态推理模型。GThinker引入‘视觉线索重思’机制,通过视觉线索锚定推理,并迭代重释以解决不一致问题。基于此,设计两阶段训练流程:模式引导冷启动与激励强化学习。为支持训练,构建了包含7000条高质量迭代标注推理路径和4000条精选强化学习样本的GThinker-11K数据集。大量实验表明,GThinker在挑战性基准M$^3$CoT上达到81.5%准确率,优于最新O4-mini模型;在通用场景多模态推理任务上平均提升2.1%,同时在数学推理方面保持与先进模型相当的性能。代码、模型与数据将很快发布于https://github.com/jefferyZhan/GThinker。
原文摘要 · Abstract (English)
Despite notable advancements in multimodal reasoning, leading Multimodal Large Language Models (MLLMs) still underperform on vision-centric multimodal reasoning tasks in general scenarios. This shortfall stems from their predominant reliance on logic- and knowledge-based slow thinking strategies, while effective for domains like math and science, fail to integrate visual information effectively during reasoning. Consequently, these models often fail to adequately ground visual cues, resulting in suboptimal performance in tasks that require multiple plausible visual interpretations and inferences. To address this, we present GThinker (General Thinker), a novel reasoning MLLM excelling in multimodal reasoning across general scenarios, mathematics, and science. GThinker introduces Cue-Rethinking, a flexible reasoning pattern that grounds inferences in visual cues and iteratively reinterprets these cues to resolve inconsistencies. Building on this pattern, we further propose a two-stage training pipeline, including pattern-guided cold start and incentive reinforcement learning, designed to enable multimodal reasoning capabilities across domains. Furthermore, to support the training, we construct GThinker-11K, comprising 7K high-quality, iteratively-annotated reasoning paths and 4K curated reinforcement learning samples, filling the data gap toward general multimodal reasoning. Extensive experiments demonstrate that GThinker achieves 81.5% on the challenging comprehensive multimodal reasoning benchmark M$^3$CoT, surpassing the latest O4-mini model. It also shows an average improvement of 2.1% on general scenario multimodal reasoning benchmarks, while maintaining on-par performance in mathematical reasoning compared to counterpart advanced reasoning models. The code, model, and data will be released soon at https://github.com/jefferyZhan/GThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。