让大模型在无监督下自我改进推理能力，不靠标签也能优化思考过程。
iReasoner: Trajectory-Aware Intrinsic Reasoning Supervision for Self-Evolving Large Multimodal Models
- 通过追问中间思考步骤，用内部一致性来奖励推理路径。
- 在多个多模态推理任务上，无监督训练提升2.1分，效果显著。
- 适合研究自进化模型与无监督推理优化的学者使用。
近期研究表明，大型多模态模型（LMMs）可通过自对弈和内在反馈从无标签数据中自我改进。然而，现有自进化框架主要奖励最终结果，导致中间推理过程缺乏有效约束，尽管其对视觉引导决策至关重要。我们提出iReasoner，一种自进化框架，通过显式激发思维链（CoT）并奖励其内部一致性，增强LMM的隐式推理能力。在无标签图像上的提议者-求解器循环中，iReasoner将结果级内在奖励扩展为轨迹感知信号，基于中间推理步骤提供学习信号，可在无真实标签或外部裁判的情况下区分达成相同答案的不同推理路径。从Qwen2.5-VL-7B出发，iReasoner在完全无监督后训练下，跨多个多模态推理基准最高提升2.1分。我们希望这项工作能成为纯无监督环境下多模态模型推理感知自改进的起点。代码已公开于https://meghanaasunil.github.io/iReasoner。
原文摘要 · Abstract (English)
Recent work shows that large multimodal models (LMMs) can self-improve from unlabeled data via self-play and intrinsic feedback. Yet existing self-evolving frameworks mainly reward final outcomes, leaving intermediate reasoning weakly constrained despite its importance for visually grounded decision making. We propose iReasoner, a self-evolving framework that improves an LMM's implicit reasoning by explicitly eliciting chain-of-thought (CoT) and rewarding its internal agreement. In a Proposer--Solver loop over unlabeled images, iReasoner augments outcome-level intrinsic rewards with a trajectory-aware signal defined over intermediate reasoning steps, providing learning signals that distinguish reasoning paths leading to the same answer without ground-truth labels or external judges. Starting from Qwen2.5-VL-7B, iReasoner yields up to $+2.1$ points across diverse multimodal reasoning benchmarks under fully unsupervised post-training. We hope this work serves as a starting point for reasoning-aware self-improvement in LMMs in purely unsupervised settings. Our code is available at https://meghanaasunil.github.io/iReasoner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。