ADPO通过锚定对数概率，统一优化策略与偏好，提升模型稳定性与泛化能力。
ADPO: Anchored Direct Preference Optimization
- 以对数比值为优化目标，用锚定对数参数化策略，解耦响应质量与先验流行度。
- 在Qwen3-1.7B上比GRPO提升30.9%，且在分布外数据下表现更稳健。
- 适合需要稳定训练和强泛化能力的对话系统与推理任务部署。
我们提出锚定直接偏好优化（ADPO），一种基于KL正则化强化学习原理的策略对齐方法。不同于将参考策略仅视为正则项的标准方法，我们表明人类反馈强化学习中的最优策略本质上运行于微分坐标系中，通过优化对数比值形式的相对优势而非绝对概率。ADPO通过锚定对数显式参数化该最优结构，有效解耦响应质量与先验流行度，并通过曲率缩放构建隐式信任区域。该公式统一了监督微调、强化学习与基于排序的目标，从单一几何视角出发。理论上，ADPO解决了监督微调的概率弥散问题，同时避免了反向KL方法的模式聚焦不稳定性。实证上，其列表排序变体在推理任务中达到当前最佳性能，在Qwen3-1.7B上相比GRPO提升30.9%，并在分布偏移下展现出更强鲁棒性。
原文摘要 · Abstract (English)
We present Anchored Direct Preference Optimization (ADPO), a policy alignment method derived from first principles of KL-regularized reinforcement learning. Unlike standard approaches that treat the reference policy merely as a regularizer, we show that the optimal policy in reinforcement learning from human feedback inherently operates in a differential coordinate system, optimizing relative advantage in the form of log ratios rather than absolute probabilities. ADPO explicitly parameterizes this optimal structure through anchored logits, effectively decoupling response quality from prior popularity and creating an implicit trust region through curvature scaling. We show that this formulation unifies supervised fine-tuning, reinforcement learning, and ranking-based objectives under a single geometric perspective. Theoretically, ADPO resolves the probability smearing problem of supervised fine-tuning while avoiding the mode-seeking instability characteristic of reverse-KL methods. Empirically, the listwise ranking variant of ADPO achieves state-of-the-art performance on reasoning tasks, outperforming GRPO by 30.9 percent on Qwen3-1.7B and demonstrating superior robustness under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。