让奖励模型适应多阶段生成流程，提升代码解题准确率。
PARM: Pipeline-Adapted Reward Model
- 用流水线数据和偏好优化训练奖励模型，对齐真实执行结果。
- 在4个优化基准上，执行成功率与求解准确率均优于基线方法。
- 适合需要多步推理的复杂任务，如算法生成与自动求解。
奖励模型（RMs）在对齐大语言模型（LLMs）与人类偏好方面至关重要，广泛用于强化学习人类反馈（RLHF）和高级解码策略。尽管以往研究集中于单步生成，但现实应用越来越多采用多阶段LLM流水线，而有效奖励引导仍缺乏探索。本文以组合优化的代码生成为例，构建融合奖励模型的两阶段流水线（建模 -> 代码生成），发现奖励预测与实际执行结果存在显著不一致。为此提出流水线自适应奖励模型（PARM），利用流水线特异性数据并采用直接偏好优化，使奖励与下游反馈对齐。在四个公开优化基准上评估，对比基线与采样方法，测量执行率与求解准确率。另通过跨领域实验在GSM8K上验证迁移能力。结果表明，PARM持续提升流水线输出质量与稳定性，为多阶段大模型推理中的奖励建模提供新洞见。
原文摘要 · Abstract (English)
Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering RLHF and advanced decoding strategies. While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage LLM pipelines, where effective reward guidance remains underexplored. We investigate this through code generation for combinatorial optimization, constructing a pipeline that integrates reward models into both formulation and solution stages. We identify a critical challenge: inconsistency between reward model predictions and actual pipeline execution outcomes. To address this, we propose the Pipeline-Adapted Reward Model (PARM), which leverages pipeline-specific data and direct preference optimization to align rewards with downstream feedback. We instantiate PARM as a two-stage pipeline (formulation -> code generation) and evaluate it on four public optimization benchmarks, measuring execution rate and solving accuracy against baselines and sampling methods. A supplementary cross-domain experiment on GSM8K assesses transferability. Results demonstrate that PARM consistently improves pipeline output quality and stability, providing new insights into reward modeling for multi-stage LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。