针对迭代偏好优化中生成过长回复的问题，提出新训练目标提升对齐效果。
AIPO: Improving Training Objective for Iterative Preference Optimization
- 提出AIPO方法，通过关注回复一致性改进迭代偏好优化
- 在MT-Bench、AlpacaEval 2.0和Arena-Hard上达到最新最好性能
- 适合追求高效对齐且需控制输出长度的LLM训练场景
偏好优化（PO）作为大型语言模型（LLMs）对齐的替代方案，正逐步取代近端策略优化（PPO）。近期研究显示，使用合成或部分合成数据进行迭代式偏好优化，在学术与私有模型（如Llama3）训练中均表现出良好扩展性。然而，我们的研究表明，由于迭代过程的特性，迭代偏好优化中的长度利用问题比传统PO更为严重。本文系统研究了基于合成数据的迭代偏好优化流程，揭示了其存在的长度利用问题，并提出新的训练目标——一致性感知的迭代偏好优化（Agreement-aware Iterative Preference Optimization, AIPO）。为验证方法有效性，我们进行了全面实验，在MT-Bench、AlpacaEval 2.0和Arena-Hard上均取得当前最优表现。代码与模型检查点将公开于https://github.com/bytedance/AIPO。
原文摘要 · Abstract (English)
Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at https://github.com/bytedance/AIPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。