提出稳定高效的隐空间推理优化方法,显著缩短推理链并提升准确率。
Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning

- 基于分组相对策略优化,设计三重机制应对隐空间推理挑战。
- 低难度任务提效7.86点,高难度任务超越显式方法4.27点,推理链缩短3-4倍。
- 适合追求高效推理与模型压缩的开发者及研究者。
隐空间推理通过将中间推理过程压缩为连续表示,大幅缩短推理链,实现更高效推理。然而,现有方法多依赖监督学习,隐空间强化学习仍极不稳定。本文从分组相对策略优化(GRPO)视角研究该问题,发现直接套用GRPO于隐空间存在根本性挑战:隐空间改变概率密度与采样机制,导致三大耦合瓶颈——无效隐空间流形缺失、探索与优化错位、隐空间混合非闭合。为此,我们提出Latent-GRPO,融合无效样本优势掩码、单侧噪声采样与最优正确路径首标记选择。在四个低难度基准(如GSM8K-Aug)和四个高难度基准(如AIME)上,相比初始隐空间方法,低难度任务提升7.86 Pass@1点,高难度任务超越显式GRPO 4.27点,同时推理链缩短3–4倍。在Gumbel采样下,其Pass@$k$性能也更优。结果验证了该方法在稳定性和效率上的有效性。
原文摘要 · Abstract (English)
Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose \textbf{Latent-GRPO}, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4$\times$ shorter reasoning chains. It also achieves stronger pass@$k$ performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。