让每个词的优化权重随训练动态调整，提升语言模型对齐效果。
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
- 基于模型内部信号动态计算每个词的贡献权重
- 在AlpacaEval2和Arena-Hard上分别提升9.8和12.2分
- 无需外部模型，仅增加轻量校准网络
直接偏好优化（DPO）通过均匀求和的方式聚合词级对数概率比，隐式地将所有词视为对偏好信号的贡献相同。然而，各词对偏好信号的实际贡献存在差异。本文引入‘词信用’机制，根据每词对偏好结果的贡献程度调节其KL正则化强度。推导表明，有效的词信用与每词隐含奖励的幅度成正比，并观察到该值在训练过程中显著变化。这说明静态词信用会随训练进展逐渐失准。为此，本文提出Se-DPO（自演化词信用），一种在DPO训练中实时从模型自身演化信号中提取词信用的机制。由于不同位置的奖励信号可靠性不同，Se-DPO结合贡献强度与置信度进行校准。该方法无需外部模型，仅引入一个轻量级校准网络，计算开销极小。实验显示，Se-DPO在AlpacaEval~2上相比DPO最高提升9.8分，在Arena-Hard上提升12.2分。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。