APO统一调控强化学习中的模式覆盖与探索，稳定提升模型表现。
APO: Alpha-Divergence Preference Optimization
- 基于锚点坐标系，用alpha参数连续切换前向与反向KL行为。
- 在数学任务上性能媲美GRPO/GSPO，训练过程更稳定。
- 适合追求稳定训练且需平衡覆盖与探索的对齐场景。
现代对齐方法主要依赖两种散度范式。监督微调和多数蒸馏目标隐式最小化前向KL散度KL(q || pi_theta)，更新稳定但常低估高奖励模式；而类似PPO的在线强化学习则趋近于反向KL散度KL(pi_theta || q)，可实现模式搜索但易导致模式崩溃。近期锚定方法（如ADPO）表明，在锚定坐标系中投影能显著提升稳定性，但通常固定使用单一散度。本文提出Alpha-Divergence Preference Optimization (APO)，一个在相同锚定几何下，通过Csiszar alpha散度连续插值前向与反向KL行为的锚定框架。我们推导出由alpha参数化的统一梯度动态，分析梯度方差特性，并提出一种基于奖励与置信度的alpha调度策略，仅在策略既提升又可靠校准时才从覆盖转向利用。在Qwen3-1.7B模型上进行数学级别3的任务实验表明，APO在保持训练稳定性的同时，性能与GRPO、GSPO基线相当。
原文摘要 · Abstract (English)
Two divergence regimes dominate modern alignment practice. Supervised fine-tuning and many distillation-style objectives implicitly minimize the forward KL divergence KL(q || pi_theta), yielding stable mode-covering updates but often under-exploiting high-reward modes. In contrast, PPO-style online reinforcement learning from human feedback behaves closer to reverse KL divergence KL(pi_theta || q), enabling mode-seeking improvements but risking mode collapse. Recent anchored methods, such as ADPO, show that performing the projection in anchored coordinates can substantially improve stability, yet they typically commit to a single divergence. We introduce Alpha-Divergence Preference Optimization (APO), an anchored framework that uses Csiszar alpha-divergence to continuously interpolate between forward and reverse KL behavior within the same anchored geometry. We derive unified gradient dynamics parameterized by alpha, analyze gradient variance properties, and propose a practical reward-and-confidence-guarded alpha schedule that transitions from coverage to exploitation only when the policy is both improving and confidently calibrated. Experiments on Qwen3-1.7B with math-level3 demonstrate that APO achieves competitive performance with GRPO and GSPO baselines while maintaining training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。