让隐式推理模型通过奖励学习实现更长思考，提升难题解答能力。
SLPO: Scaling Latent Reasoning via a Surrogate Policy
- 用代理策略密度估算潜空间转移概率，实现全程奖励分配。
- 在并行采样下，通过更长的潜空间计算，提升Pass@$k$指标。
- 自动识别难例并延长推理时间，适合高精度需求场景。
基于可验证奖励的强化学习已成为激发显式思维链推理模型测试时扩展性的主流方法。然而，这种扩展路径计算成本高昂，因为每个中间步骤都需解码为语言符号。隐式推理则将中间计算表示为连续向量，在更短的推理步数下已达到或超越显式思维链的表现。尽管前景广阔，隐式推理仍受限于模仿学习，而显式思维链已通过结果奖励强化学习超越模仿。由于隐式轨迹缺乏可计算的每步似然和固定思考预算下的自适应停止机制，无法有效利用结果奖励进行测试时扩展。本文提出代理隐式策略优化（SLPO），将结果奖励强化学习引入自回归隐式推理：设计经验性代理策略密度以实现轨迹级信用分配，并引入正确性监督的停止头，使结果奖励优化将其转化为可变长度推理策略。在连续与软思考设置中，SLPO 在并行采样下提升了 Pass@$k$，并将更长的潜空间计算分配给更难的问题，获得更高的确定性准确率。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@$k$ under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。