根据模型置信度自动拆分推理步骤，提升奖励模型性能并降低成本
AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
- 基于模型预测下一词的置信度动态划分推理步骤
- 在数学推理与代码生成任务中达到顶尖的Best-of-N表现
- 无需人工标注，构建成本降低30%以上，适合高效训练奖励模型
当前训练过程奖励模型（PRM）的方法多依赖规则化手段将回答拆分为多个推理步骤，如使用预设占位符或固定长度划分。这些方法忽视了文本中特定词汇通常不标志真正的决策点。为此，我们提出AdaptiveStep，依据模型对下一个词的预测置信度来划分推理步骤。该方法在每一步提供更丰富的决策信息，有助于下游任务如奖励模型学习。此外，该方法无需人工标注。我们在数学推理和代码生成任务中验证其有效性，实验表明，采用AdaptiveStep训练的PRM在Best-of-N任务中表现优于贪婪搜索策略与基于分词值引导的解码方法，同时构建成本比现有开源PRM降低超过30%。我们还对PRM的性能、可迁移性与泛化能力进行了深入分析与案例研究。
原文摘要 · Abstract (English)
Current approaches for training Process Reward Models (PRMs) often involve breaking down responses into multiple reasoning steps using rule-based techniques, such as using predefined placeholder tokens or setting the reasoning step's length into a fixed size. These approaches overlook the fact that specific words do not typically mark true decision points in a text. To address this, we propose AdaptiveStep, a method that divides reasoning steps based on the model's confidence in predicting the next word. This division method provides more decision-making information at each step, enhancing downstream tasks, such as reward model learning. Moreover, our method does not require manual annotation. We demonstrate its effectiveness through experiments with AdaptiveStep-trained PRMs in mathematical reasoning and code generation tasks. Experimental results indicate that the outcome PRM achieves state-of-the-art Best-of-N performance, surpassing greedy search strategy with token-level value-guided decoding, while also reducing construction costs by over 30% compared to existing open-source PRMs. In addition, we provide a thorough analysis and case study on the PRM's performance, transferability, and generalization capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。