提出风险感知的偏好优化方法，更好控制模型偏离参考行为的风险。
Risk-aware Direct Preference Optimization under Nested Risk Measure
- 用嵌套风险度量构建风险感知的优势函数优化
- 在三个数据集上同时提升对齐效果与降低模型漂移
- 适合需要严格风险控制的高阶应用
微调预训练大语言模型以对齐人类价值观和意图时，单纯最大化估计奖励虽能提升性能，但可能因偏离参考模型的预期行为引入潜在风险。现有方法多通过KL散度约束模型与参考模型的差异，但在需严格风险控制的应用中仍显不足。本文提出风险感知直接偏好优化（Ra-DPO），引入一类嵌套风险度量，将布莱特利-泰勒模型转化为逐标记级表示，构建受限的风险感知优势函数最大化问题。该方法在最大化策略似然的同时，利用序列风险比抑制训练模型与参考模型间的偏差，增强风险感知能力。在IMDb、Anthropic HH和AlpacaEval三个开源数据集上的实验表明，所提方法在对齐性能与模型漂移之间实现更优平衡。代码已开源：https://github.com/zlj123-max/Ra-DPO。
原文摘要 · Abstract (English)
When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead to superior performance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KL divergence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures. This approach formulates a constrained risk-aware advantage function maximization problem and then converts the Bradley-Terry model into a token-level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using a sequential risk ratio, thereby enhancing the model's risk-awareness. Experimental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's superior performance in balancing alignment performance and model drift. Our code is opensourced at https://github.com/zlj123-max/Ra-DPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。