LPO通过梯度解耦提升对齐模型稳定性，防止过拟合与崩溃。
Linear Preference Optimization: Decoupled Gradient Control via Absolute Regularization
- 用绝对差损失替代log-sigmoid，实现梯度解耦
- 引入偏移约束与正则项，保持优选响应质量
- 可调系数控制拒绝概率下降，适合需要精细调控的场景
DPO（直接偏好优化）因简洁和训练稳定被广泛采用，但易出现过拟合与崩溃。为此，我们提出线性偏好优化（LPO），包含三项关键创新：首先，通过用绝对差损失替代log-sigmoid函数实现梯度解耦，分离优化动态；其次，结合偏移约束与正则项提升稳定性，保障优选响应质量；第三，利用梯度分离与可调系数实现可控拒绝抑制，支持线性调节拒绝概率下降。大量实验表明，LPO在通用文本、数学推理及文本转语音（TTS）任务中均持续提升性能。该方法构建了一个稳健且可调的偏好对齐范式，代码、模型与训练数据已公开。
原文摘要 · Abstract (English)
DPO (Direct Preference Optimization) has become a widely used offline preference optimization algorithm due to its simplicity and training stability. However, DPO is prone to overfitting and collapse. To address these challenges, we propose Linear Preference Optimization (LPO), a novel alignment framework featuring three key innovations. First, we introduce gradient decoupling by replacing the log-sigmoid function with an absolute difference loss, thereby isolating the optimization dynamics. Second, we improve stability through an offset constraint combined with a positive regularization term to preserve the chosen response quality. Third, we implement controllable rejection suppression using gradient separation with straightforward estimation and a tunable coefficient that linearly regulates the descent of the rejection probability. Through extensive experiments, we demonstrate that LPO consistently improves performance on various tasks, including general text tasks, math tasks, and text-to-speech (TTS) tasks. These results establish LPO as a robust and tunable paradigm for preference alignment, and we release the source code, models, and training data publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。