为多模态推理设计细粒度策略优化，提升视觉与推理结合能力。
Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
- 基于隐藏状态相似性构建感知先验，结合熵平滑门控实现逐标记优势计算
- 在几何推理、视觉谜题等任务上显著超越强基线，提升幅度达3.5%~12.7%
- 无需额外监督或分支，可无缝集成现有强化学习框架，适合多模态推理研究者
多模态链式思维（CoT）推理要求大视觉语言模型在感知锚定与多步推理之间交替生成推理轨迹。然而，现有可验证奖励强化学习（RLVR）方法通常以粗粒度优化推理过程，未区分不同推理步骤的视觉锚定程度。本文对多模态推理轨迹进行逐标记分析，发现成功推理具有反映感知锚定与探索性推理的结构化标记动态。基于此，提出感知-探索策略优化（PEPO），通过隐藏状态相似性构建感知先验，并以平滑门控机制将其与标记熵结合，生成逐标记优势。PEPO可无缝集成于GRPO、DAPO等现有RLVR框架，无需额外监督或辅助分支。在多种多模态基准上广泛实验表明，其在几何推理、视觉锚定、视觉解谜和少样本分类任务中均实现稳定且一致的性能提升，绝对改进达3.5%~12.7%，同时保持训练稳定性。
原文摘要 · Abstract (English)
Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories that interleave perceptual grounding with multi-step inference. However, existing Reinforcement Learning with Verifiable Rewards (RLVR) methods typically optimize reasoning at a coarse granularity, treating CoT uniformly without distinguishing their varying degrees of visual grounding. In this work, we conduct a token-level analysis of multimodal reasoning trajectories and show that successful reasoning is characterized by structured token dynamics reflecting both perceptual grounding and exploratory inference. Building upon this analysis, we propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior from hidden state similarity and integrates it with token entropy through a smooth gating mechanism to produce token-level advantages. PEPO integrates seamlessly with existing RLVR frameworks such as GRPO and DAPO, requiring neither additional supervision nor auxiliary branches. Extensive experiments across diverse multimodal benchmarks demonstrate consistent and robust improvements over strong RL baselines, spanning geometry reasoning, visual grounding, visual puzzle solving, and few-shot classification, while maintaining stable training dynamics. Code: https://github.com/xzxxntxdy/PEPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。