用教师模型的奖励值引导小模型对齐人类偏好，提升训练效果。
Preference Distillation via Value based Reinforcement Learning
- 引入教师模型的价值函数作为辅助奖励，提供软性指导。
- 在多个基准上提升不同规模模型的对齐性能，效果稳定。
- 无需额外采样，可无缝融入标准DPO训练流程，适合小模型优化。
直接偏好优化（DPO）是一种利用成对比较对齐语言模型与人类偏好的有效方法。然而，其二元胜负监督对于容量有限的小模型常显不足。以往工作尝试通过行为克隆或KL散度从大模型中蒸馏信息，但多聚焦于模仿当前行为，忽视了奖励建模的蒸馏。为此，我们提出教师价值基础知识蒸馏（TVKD），引入教师模型价值函数的辅助奖励，提供软性指导。该奖励满足基于潜在的奖励重塑，确保DPO的全局奖励结构和最优策略得以保留。TVKD可集成至标准DPO训练框架，无需额外采样。实验结果表明，TVKD在多个基准和模型尺寸下均持续提升性能。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) is a powerful paradigm to align language models with human preferences using pairwise comparisons. However, its binary win-or-loss supervision often proves insufficient for training small models with limited capacity. Prior works attempt to distill information from large teacher models using behavior cloning or KL divergence. These methods often focus on mimicking current behavior and overlook distilling reward modeling. To address this issue, we propose \textit{Teacher Value-based Knowledge Distillation} (TVKD), which introduces an auxiliary reward from the value function of the teacher model to provide a soft guide. This auxiliary reward is formulated to satisfy potential-based reward shaping, ensuring that the global reward structure and optimal policy of DPO are preserved. TVKD can be integrated into the standard DPO training framework and does not require additional rollouts. Our experimental results show that TVKD consistently improves performance across various benchmarks and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。