用价值模型指导推理搜索，提升长文本推理效率与效果。
Value-Guided Search for Efficient Chain-of-Thought Reasoning
- 基于250万条推理轨迹训练15亿参数值函数模型，无需精细步骤标注。
- 块级价值引导搜索在测试时计算量更少，性能优于多数投票和最佳N法。
- 适合需要高效推理的长文本任务，如大模型数学与逻辑推理。
本文提出一种简单高效的长上下文推理轨迹值模型训练方法。相较于现有过程奖励模型（PRM），该方法无需定义复杂的“步骤”概念，适用于长上下文推理场景。通过收集250万条推理轨迹，训练了一个15亿参数的分词级别值模型，并应用于DeepSeek模型，实现测试时计算资源扩展下的性能提升。实验表明，采用块级价值引导搜索（VGS）并结合最终加权多数投票，相比标准方法如多数投票或best-of-n，在测试时具有更好的扩展性；同时，显著降低达成相同性能所需的推理浮点运算量（FLOPs）。相关数据集、模型与代码均已开源。
原文摘要 · Abstract (English)
In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。