解决大模型对齐中的排名偏差问题，提升训练效率与效果
AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
- 根据样本排序正确性动态调整梯度权重，避免无效优化
- 在多个基准上实现更高排名准确率和下游任务性能
- 适合追求高效、稳定对齐的大模型训练团队
离线偏好优化为语言模型对齐提供了比强化学习人类反馈（RLHF）更简单稳定的替代方案。然而，其效果高度依赖排名准确性，而该指标的进一步提升具有重大影响。我们识别并形式化了这一瓶颈的根本原因：现有间隔设计导致模型对正确排序样本施加过度且浪费的梯度（过拟合），同时对错误排序样本提供不足的修正信号（欠拟合）。为此，我们提出自适应间隔附着偏好优化（AMaPO），一种简洁而原理严谨的算法。AMaPO采用实例级自适应间隔，经Z标准化和指数缩放优化，动态重新分配学习资源，放大错误排序样本的梯度，抑制正确排序样本的梯度。广泛实验表明，AMaPO不仅显著提升排名准确率和下游对齐性能，且针对性分析证实其有效缓解了核心过拟合与欠拟合问题。
原文摘要 · Abstract (English)
Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models. However, their effectiveness is critically dependent on ranking accuracy, a metric where further gains are highly impactful. This limitation arises from a fundamental problem that we identify and formalize as the Overfitting-Underfitting Dilemma: current margin designs cause models to apply excessive, wasteful gradients to correctly ranked samples (overfitting) while providing insufficient corrective signals for misranked ones (underfitting). To resolve this dilemma, we propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled algorithm. AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones. Extensive experiments on widely used benchmarks demonstrate that AMaPO not only achieves better ranking accuracy and superior downstream alignment performance, but targeted analysis also confirms that it successfully mitigates the core overfitting and underfitting issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。