让大模型学会识别推理中的关键步骤，提升逻辑可靠性。
Offline-Online Curriculum RL for Multimodal Reasoning
- 通过离线分析多轮推理轨迹，识别每步重要性
- 在线阶段用分步强化学习，让模型补全缺失关键步骤
- 适合需要可解释推理的多模态任务应用
多模态大语言模型在推理任务中虽能给出正确答案，但常存在错误中间步骤，影响可解释性和可靠性，暗示其依赖表面捷径而非真实推理。尽管已有研究尝试步骤级监督，但区分决定性步骤与冗余步骤仍具挑战。我们提出 $O^2$-CritiCuRL，一种基于迭代离线-在线范式的课程强化学习框架，引入关键步骤感知机制。离线阶段，对带步骤标注的轨迹进行多轮分析，估计各步骤重要性，从而提炼关键推理路径并剔除冗余步骤；在线阶段，采用渐进式步骤级强化学习策略，通过截断推理链引导模型推断缺失步骤并优化推理过程，增强对关键步骤的关注，克服静态监督的局限。在多个多模态推理基准上的大量实验表明，该方法实现当前最优性能，并具备更优的训练与推理效率。代码已公开于 https://github.com/kk0013/CritiCuRL。
原文摘要 · Abstract (English)
Multimodal large language models exhibit capabilities on reasoning tasks, yet often produce flawed intermediate steps while yielding correct final answers. This behavior undermines interpretability and reliability, suggesting reliance on spurious shortcuts rather than faithful reasoning. Although efforts have explored step-level supervision, distinguishing decisive steps from redundant ones remains challenging. We propose $O^2$-CritiCuRL, a novel curriculum reinforcement learning framework that introduces critical-step awareness through an iterative offline-online paradigm. In the offline stage, $O^2$-CritiCuRL conducts multi-rollout analysis over step-annotated trajectories to estimate step-level importance, allowing the framework to distill critical reasoning steps and filter out redundant ones. In the online stage, we employ a progressive step-level reinforcement learning strategy, where truncated chains guide the model to infer missing steps and refine its reasoning, thereby sharpening its focus on critical steps and overcoming the limitations of static supervision. Extensive experiments on multimodal reasoning benchmarks show that our method achieves state-of-the-art performance while delivering superior training and inference efficiency. Code is available at https://github.com/kk0013/CritiCuRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。