让大模型自己生成最优折衷解，解决多目标对齐中的偏好冲突问题。
Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment
- 通过自生成与筛选帕累托最优响应，化解多目标间的冲突优化
- 在两个数据集上实现优于基线的帕累托前沿性能
- 适合研究多目标对齐与大模型自我改进的学者使用
多目标对齐（MOA）旨在使大语言模型的输出符合多个用户偏好目标，直接偏好优化（DPO）是其中主流方法。然而我们发现，基于DPO的MOA方法普遍存在数据层面的偏好冲突——不同目标倾向于不同回答，导致优化方向相互矛盾，阻碍模型在帕累托前沿上的优化。为解决此问题，我们提出构建帕累托最优响应以化解偏好冲突。为进一步高效获取和利用此类响应，我们设计了一种自提升DPO框架，使模型能自我生成并选择帕累托最优响应，实现自监督偏好对齐。在两个数据集上的大量实验表明，该框架所达到的帕累托前沿显著优于多种基线方法。代码已公开于 https://github.com/zyttt-coder/SIPO。
原文摘要 · Abstract (English)
Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines. Code is available at https://github.com/zyttt-coder/SIPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。