arXiv:2601.17275cs.LGcs.AI2026-01

用潜在空间对比强化学习,让大模型推理更稳定高效。

Latent-Space Contrastive Reinforcement Learning for Stable and Efficient LLM Reasoning

  • 在连续潜在空间而非离散文本空间进行强化学习
  • 冻结主模型实现稳定训练,支持更长推理链
  • 适合需要可靠推理能力的复杂任务研究者

尽管大型语言模型在表面文本生成上表现优异,但在处理复杂多步推理任务时,其本质仍依赖于统计拟合而非系统性逻辑推导。传统强化学习虽引入了‘先思考后表达’范式,但直接在高维离散的词元空间应用面临采样效率低、梯度估计方差大及灾难性遗忘三大挑战。为此,我们提出深度潜在推理(DeepLatent Reasoning, DLR)框架,将试错成本从昂贵的完整序列生成转移到连续潜在流形上。通过轻量级助手模型在潜在空间高效采样 $K$ 条推理链编码,再基于正确性和格式双重奖励机制筛选高价值轨迹,仅将优质潜变量输入冻结的主模型进行单次解码。为提升推理多样性同时保持连贯性,设计对比学习目标以引导潜在空间探索。由于主模型参数在优化过程中保持冻结,该方法在数学上消除了灾难性遗忘。实验表明,在相当的GPU计算预算下,DLR实现更稳定的训练收敛,支持更长推理链,并促进推理能力可持续积累,为大模型的可靠与可扩展强化学习提供可行路径。

原文摘要 · Abstract (English)

While Large Language Models (LLMs) demonstrate exceptional performance in surface-level text generation, their nature in handling complex multi-step reasoning tasks often remains one of ``statistical fitting'' rather than systematic logical deduction. Traditional Reinforcement Learning (RL) attempts to mitigate this by introducing a ``think-before-speak'' paradigm. However, applying RL directly in high-dimensional, discrete token spaces faces three inherent challenges: sample-inefficient rollouts, high gradient estimation variance, and the risk of catastrophic forgetting. To fundamentally address these structural bottlenecks, we propose \textbf{DeepLatent Reasoning (DLR)}, a latent-space bidirectional contrastive reinforcement learning framework. This framework shifts the trial-and-error cost from expensive token-level full sequence generation to the continuous latent manifold. Specifically, we introduce a lightweight assistant model to efficiently sample $K$ reasoning chain encodings within the latent space. These encodings are filtered via a dual reward mechanism based on correctness and formatting; only high-value latent trajectories are fed into a \textbf{frozen main model} for single-pass decoding. To maximize reasoning diversity while maintaining coherence, we design a contrastive learning objective to enable directed exploration within the latent space. Since the main model parameters remain frozen during optimization, this method mathematically eliminates catastrophic forgetting. Experiments demonstrate that under comparable GPU computational budgets, DLR achieves more stable training convergence, supports longer-horizon reasoning chains, and facilitates the sustainable accumulation of reasoning capabilities, providing a viable path toward reliable and scalable reinforcement learning for LLMs.

强化学习大模型推理潜在空间对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。