对比51种对齐算法发现，模型规模改变会导致最佳算法反转。
Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions
- 统一框架下对比51种后训练算法，控制变量评估性能差异。
- 1.5B模型中表现最好的算法在7B时变为最差，排名完全反转。
- 算法选择主要影响训练分布内任务，跨领域效果差异极小。
后训练对齐已发展出数十种算法（如DPO、SimPO、KTO、GRPO等），但缺乏可控比较指导实践。本文提出OXRL框架，统一实现51种算法，首次进行大规模直接对比。研究覆盖8种算法、4个模型规模（0.5B–7B）、3个评估领域，包含20种DPO变体（1.5B模型上100次运行，每种子5次随机种子），总计约240次训练，使用H100 GPU。三大发现：(1) 算法排名随模型规模变化不稳定——1.5B时在线强化学习（SGRPO）在GSM8K上达58.0%±0.57；到7B时，原最差的SimPO反成最优（85.8%），出现完全排名反转，且非由LoRA正则化导致（经2×2因子实验验证）；(2) 损失函数改进几乎无效——20种DPO变体无一在贝叶斯校正后显著优于原始DPO，唯一显著异常为更差的SimPO（-11.5pp，p<10⁻⁴）；(3) 算法效能具有任务特异性：在GSM8K上19.3pp差距缩小至MATH仅0.54pp（36倍）、通用基准0.47pp（41倍），说明算法选择主要影响训练分布内任务。给出实用优先级：模型规模（~50pp）≫训练范式（~10pp）≫在线/离线（~9pp）≫损失函数（~1pp）。所有代码、配置与数据公开，作为持续更新的社区基准。
原文摘要 · Abstract (English)
Post-training alignment has produced dozens of competing algorithms -- DPO, SimPO, KTO, GRPO, and others -- yet practitioners lack controlled comparisons to guide algorithm selection. We present OXRL, a unified framework implementing 51 post-training algorithms with identical infrastructure, enabling the first large-scale apples-to-apples evaluation. Our study spans 8 algorithms across 4 model scales (0.5B--7B), 3 evaluation domains, and a 20-variant DPO taxonomy (100 runs at 1.5B, 5 seeds each), totaling $\sim$240 training runs on H100 GPUs. Three headline findings emerge. (1)~Algorithm rankings are unstable across scale: at 1.5B, online RL (SGRPO) tops all methods at 58.0\%~$\pm$0.57 on GSM8K; by 7B, the worst small-scale method (SimPO) becomes the best (85.8\%), a complete ranking inversion driven by model scale rather than LoRA regularization (confirmed via 2$\times$2 factorial). (2)~Loss function modifications yield negligible gains: none of 20 DPO variants significantly outperform vanilla DPO after Bonferroni correction; the sole significant outlier, SimPO, is worse ($-$11.5~pp, $p < 10^{-4}$). (3)~Algorithm leverage is task-specific: the 19.3~pp GSM8K spread collapses to 0.54~pp on MATH ($36\times$) and 0.47~pp on general-domain benchmarks ($41\times$), confirming that algorithm choice matters primarily within the training distribution. These findings yield a hierarchy of leverage for practitioners: model scale (${\sim}$50~pp) $\gg$ training paradigm (${\sim}$10~pp) $\gg$ online vs.\ offline (${\sim}$9~pp) $\gg$ loss function (${\sim}$1~pp). We release all code, configs, and evaluation data as a living community benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。