提出SofT-GRPO，让大模型软思考比传统离散推理更高效。
SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization
- 用Gumbel重参数化实现软思维策略优化
- 软思考在Pass@32上比离散推理高2.19%
- 适合想提升模型推理能力的研究者
大语言模型的软思考范式在某些场景下优于传统的离散令牌思维链（CoT）推理，具有重要研究价值。然而，将强化学习（RL）应用于软思考仍面临挑战，主要源于如何向软思考令牌注入随机性并有效更新策略。此前结合软思考与组相对策略优化（GRPO）的方法通常表现逊于离散令牌版本。本文提出新型策略优化算法SofT-GRPO，通过在logits中注入Gumbel噪声，利用Gumbel-Softmax避免软思考令牌脱离预训练嵌入空间，并结合重参数化技巧优化策略梯度。在1.5B至7B参数量的基础模型上进行实验，结果表明：SofT-GRPO使软思考模型在Pass@1上平均准确率提升0.13%，在Pass@32上平均提升2.19%。代码与权重已开源。
原文摘要 · Abstract (English)
The soft-thinking paradigm for Large Language Model (LLM) reasoning can outperform the conventional discrete-token Chain-of-Thought (CoT) reasoning in some scenarios, underscoring its research and application value. However, while the discrete-token CoT reasoning pattern can be reinforced through policy optimization algorithms such as group relative policy optimization (GRPO), extending the soft-thinking pattern with Reinforcement Learning (RL) remains challenging. This difficulty stems from the complexities of injecting stochasticity into soft-thinking tokens and updating soft-thinking policies accordingly. As a result, previous attempts to combine soft-thinking with GRPO typically underperform their discrete-token GRPO counterparts. To fully unlock the potential of soft-thinking, this paper presents a novel policy optimization algorithm, SofT-GRPO, to reinforce LLMs under the soft-thinking reasoning pattern. SofT-GRPO injects the Gumbel noise into logits, employs the Gumbel-Softmax technique to avoid soft-thinking tokens outside the pre-trained embedding space, and leverages the reparameterization trick in policy gradient. We conduct experiments across base LLMs ranging from 1.5B to 7B parameters, and results demonstrate that SofT-GRPO enables soft-thinking LLMs to slightly outperform discrete-token GRPO on Pass@1 (+0.13% on average accuracy), while exhibiting a substantial uplift on Pass@32 (+2.19% on average accuracy). Codes and weights are available on https://github.com/zz1358m/SofT-GRPO-master
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。