用强化学习让大模型混合使用隐状态与生成词元,提升推理能力。
Hybrid Latent Reasoning via Reinforcement Learning
- 通过可学习门控机制融合隐藏状态与词元,实现混合推理
- 在多个任务上超越已有方法,知识与推理类任务均表现更优
- 无需思维链轨迹,训练过程更灵活,适合追求高效推理的研究者
近期大语言模型的进展推动了隐式推理作为自回归推理的替代方案。通过利用前步隐藏状态进行内部计算,隐式推理能获得比离散思维链(CoT)路径更丰富的特征表示。然而,这类方法常与大语言模型不兼容,因其连续范式与自回归生成的离散性冲突,且依赖CoT轨迹训练,未能挖掘大模型自身的推理模式。本文提出基于强化学习的混合推理策略优化(HRPO),其一,通过可学习门控机制将先前隐藏状态融入采样词元;其二,初期以词嵌入为主,逐步引入更多隐藏特征。该设计保留了模型生成能力,并通过离散与连续表示的结合激励混合推理。此外,通过词元采样引入随机性,使基于强化学习的优化无需依赖CoT轨迹。广泛评估显示,HRPO在多类基准测试中优于现有方法,训练后的模型保持可解释性,展现出跨语言模式与更短生成长度等有趣行为,为隐式推理研究提供了新思路。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete chain-of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with LLMs, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail to exploit the inherent reasoning patterns of LLMs. In this work, we explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with a learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes hybrid reasoning using both discrete and continuous representations. In addition, the hybrid HRPO introduces stochasticity into latent reasoning via token sampling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。