用指令微调提升奖励模型对人类偏好的对齐效果
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
- 将问答对转为偏好指令，利用大模型的指令遵循能力
- 多指令平均降低任务偏差，提升评估鲁棒性
- 不同丢弃率下输出平均，稳定奖励估计
奖励模型在对齐大语言模型与人类偏好方面至关重要。现有方法存在两大局限：判别式奖励模型需大规模标注数据，无法利用生成式奖励模型中大模型的偏好指令遵循能力；同时，奖励模型易受奖励过优化影响，导致大模型利用奖励函数漏洞而非真正提升对齐度。本文提出PIRA训练范式，融合三项互补策略：（1）将问答对重构为偏好任务指令，显式利用大模型的偏好指令遵循能力；（2）对每个样本从多种偏好任务指令中聚合奖励并取平均，缓解任务特定偏差，增强跨评估视角的鲁棒性；（3）对价值头在不同丢弃率下的输出进行平均，稳定奖励估计。在公开数据集上的实验表明，PIRA显著提升性能，增强泛化能力，并有效缓解奖励过优化问题。
原文摘要 · Abstract (English)
Reward models are pivotal for aligning Large Language Models (LLMs) with human preferences. Existing approaches face two key limitations: Discriminative reward models require large-scale annotated data, as they cannot exploit the preference instruction-following capability of LLMs available to generative reward models. Moreover, reward models are particularly prone to reward overoptimization, where LLMs exploit weaknesses in the reward function instead of improving true alignment. We introduce \textbf{PIRA}, a training paradigm that integrates three complementary strategies to address these challenges: (1) reformulating question-answer pairs into preference-task instructions to explicitly leverage LLMs' preference instruction-following capability, (2) averaging the rewards aggregated from diverse preference-task instructions for each sample, which mitigates task-specific bias and enhances robustness across evaluation perspectives, and (3) averaging outputs from the value head under different dropout rates to stabilize reward estimation. Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。