无需奖励模型，动态平衡多维度偏好，提升大模型对齐效果
AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
- 用生成指标作隐式奖励，实现多目标偏好优化
- 在7B~32B模型上提升28.5%，具备良好扩展性
- 适合需要多维可控生成的场景，如内容创作与对话系统
现有大语言模型的多目标偏好对齐方法存在两大局限：难以有效平衡不同偏好维度，且依赖额外的奖励或参考模型带来计算开销。为此，我们提出自适应多目标偏好优化（AMoPO），通过将多目标优化引入生成过程，利用维度感知的生成指标作为隐式奖励，实现无需额外奖励模型或参考模型的偏好对齐。该方法设计了自适应权重分配机制，将生成空间建模为高斯分布，动态调整各偏好维度的优先级。实验表明，AMoPO相比最先进基线提升28.5%，在7B、14B和32B模型上均表现出良好的可扩展性。多维度分析进一步验证其适应性与有效性。结果表明，AMoPO能够实现维度感知的偏好对齐，具备显著优势。代码与数据集已开源。
原文摘要 · Abstract (English)
Existing multi-objective preference alignment methods for large language models (LLMs) face limitations: (1) the inability to effectively balance various preference dimensions, and (2) reliance on auxiliary reward/reference models introduces computational complexity. To address these challenges, we propose Adaptive Multi-objective Preference Optimization (AMoPO), a novel framework that achieves dynamic balance across preference dimensions. By introducing the multi-objective optimization paradigm to use the dimension-aware generation metrics as implicit rewards, AMoPO aligns LLMs with diverse preferences without additional reward models or reference models. We introduce an adaptive weight assignment mechanism that models the generation space as a Gaussian distribution, allowing dynamic prioritization of preference dimensions. Empirical results demonstrate that AMoPO outperforms state-of-the-art baselines by 28.5%, and the experiments on 7B, 14B, and 32B models reveal the scaling ability of AMoPO. Moreover, additional analysis of multiple dimensions verifies its adaptability and effectiveness. These findings validate AMoPO's capability to achieve dimension-aware preference alignment, highlighting its superiority. Our codes and datasets are available at https://github.com/Javkonline/AMoPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。