小模型也能当代码评估裁判，兼顾过程与结果评分
Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewards
- 用小模型加价值头，同时评估代码生成过程和最终结果
- 在APPS数据集上，小模型能准确识别正确代码解法
- 提升20%以上代码搜索能力，适合资源有限的场景
大型语言模型在生成高质量代码方面仍面临挑战。为推动推理模型在此任务上的发展，奖励模型作为中间步骤至关重要，用于评判输出结果或中间步骤。通过引入回归层并进行监督微调，解码器仅有的Transformer模型可转化为奖励模型。尽管反射能力通常随模型规模增大而增强，我们探究了如Phi-4系列等前沿小型语言模型是否可通过结合过程奖励与结果奖励，转化为可用的奖励模型。为此，我们构建了一个基于APPS编码挑战基准的代码样本数据集，包含正确性标签。随后训练一个价值头部模型以估计中间输出的成功概率。评估表明，小型LLM能够有效充当代码评估批评者，成功从多个候选解中识别出正确解。利用该批评者，我们在多个生成结果中寻找最准确代码的能力提升了超过20%。
原文摘要 · Abstract (English)
Generating high-quality code remains a challenge for Large Language Models (LLMs). For the evolution of reasoning models on this task, reward models are a necessary intermediate step. These models judge outcomes or intermediate steps. Decoder-only transformer models can be turned into reward models by introducing a regression layer and supervised fine-tuning. While it is known that reflection capabilities generally increase with the size of a model, we want to investigate whether state-of-the-art small language models like the Phi-4 family can be turned into usable reward models blending the consideration of process rewards and outcome rewards. Targeting this goal, we construct a dataset of code samples with correctness labels derived from the APPS coding challenge benchmark. We then train a value-head model to estimate the success probability of intermediate outputs. Our evaluation shows that small LLMs are capable of serving as effective reward models or code evaluation critics, successfully identifying correct solutions among multiple candidates. Using this critic, we achieve over a 20% improvement in the search capability of the most accurate code out of multiple generations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。