用强化学习优化大模型在隐空间中的思考过程，提升复杂推理能力。
Reinforcement Learning for Latent-Space Thinking in LLMs
- 采用强化学习直接优化隐空间中的思维步骤
- 在数学推理任务上仍落后于传统语言空间的思维方法
- 开源代码，推动隐空间推理研究
链式思维（CoT）通常在离散语言空间中进行推理，效率低下，因大量生成词元仅符合语言规则而非推理需求。为突破此限制，隐空间思考允许模型使用连续嵌入空间进行推理。现有训练方法虽在特定领域取得成效，但在复杂任务（如数学推理）中表现不佳。我们实验发现，一种名为Coconut的监督微调方法对设计选择高度敏感，存在固有缺陷。为此，我们探索了强化学习（RL）技术——这一在隐空间思考中尚未被充分研究的方向——包括GRPO，并提出一种新的隐空间强化学习方法，直接优化隐性思考步骤。实验表明，尽管如此，这些强化学习训练的模型在数学推理任务上仍不及传统的语言空间链式思维模型。代码已公开。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) reasoning typically utilizes the discrete language space for thinking, which is inherently inefficient, as many generated tokens only enforce linguistic rules that are not required for reasoning. To bypass this, latent-space thinking allows models to think using the continuous embedding space. While existing methods for training those models show domain-specific gains, they fail to maintain performance in complex tasks, such as mathematical reasoning. We experimentally demonstrate that the Coconut approach, a form of supervised fine-tuning for latent-space thinking, is highly sensitive to design choices and exhibits several inherent limitations. To address these issues, we investigate reinforcement learning (RL) techniques -- an underexplored direction in latent-space thinking -- including GRPO and design a novel Latent RL method for directly optimizing the latent thinking steps. Our experimental results reveal that these RL-trained models still lag behind traditional language-space CoT models in the mathematical reasoning domain. We make our codebase publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。