让大模型并行推理时能自动发现正确且多样化的思考路径。
Training Large Language Models To Reason In Parallel With Global Forking Tokens
- 用全局分叉标记和集合损失训练模型，保持多种有效推理路径。
- 在数学和代码生成任务上优于传统方法，提升推理能力。
- 适合需要复杂逻辑推理的研究者与开发者使用。
尽管大型语言模型通过增加测试时并行计算量提升了性能，但需生成既多样又准确的推理路径。对于难题，能触发多样且正确推理模式的分叉标记通常位于采样树深处。因此，常见鼓励多样性的策略（如温度调节）会加剧多样性和准确性之间的权衡。为此，我们将并行推理视为一组下一个标记预测问题，通过全局分叉标记与唯一推理轨迹间的二部匹配，将集合型全局损失引入监督微调（SFT）。我们发现，简单地用多条推理轨迹进行微调会导致不同推理模式坍缩，而提出的集合监督微调（SSFT）可保留这些模式，并涌现出新的全局分叉标记。全局分叉策略优化（GFPO）利用这些高度可控的标记来激励复杂推理，使模型在数学推理与基于执行的代码生成基准上均持续优于使用GRPO的SFT模型。
原文摘要 · Abstract (English)
Although LLMs have demonstrated improved performance by scaling parallel test-time compute, doing so relies on generating reasoning paths that are both diverse and accurate. For challenging problems, the forking tokens that trigger diverse yet correct reasoning modes are typically deep in the sampling tree. Consequently, common strategies to encourage diversity, such as temperature scaling, encounter a worsened trade-off between diversity and accuracy. Motivated by this challenge, we treat parallel reasoning as a set-of-next-token-prediction problem and incorporate a set-based global loss into Supervised Fine-Tuning (SFT) using bipartite matching between global forking tokens and unique reasoning traces. We observe that whereas naive fine-tuning with multiple reasoning traces collapses these unique reasoning modes, our proposed method, Set Supervised Fine-Tuning (SSFT), preserves these modes and produces emergent global forking tokens. Global Forking Policy Optimization (GFPO) leverages these maximally steerable tokens to incentivize complex reasoning, and the resulting models consistently outperform their SFT counterparts with GRPO on both math reasoning and execution-based code generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。