动态筛选有效训练样本，提升推理模型对齐效率与稳定性。
Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models
- 根据模型能力动态更新训练样本池，优先选择有信息量的错误样本。
- 在数学推理任务上收敛速度提升30%以上，性能优于传统方法。
- 适合追求高效训练和稳定优化的大型推理模型研究者使用。
基于偏好的对齐是训练大模型推理能力的关键；然而，标准方法如直接偏好优化（DPO）通常对所有偏好对一视同仁，忽略了训练样本价值的动态变化。这种静态策略常导致优化低效或不稳定，因计算资源浪费在梯度微弱的平凡样本上，并受临近不确定决策边界的噪声样本干扰。为此，我们提出SAGE（Stability-Aware Gradient Efficiency），一种通过最大化策略更新信噪比来增强对齐可靠性的动态框架。SAGE结合粗粒度课程机制，依据模型能力刷新候选样本池，同时采用细粒度的稳定性感知评分函数，优先选择有信息量且自信的错误样本，过滤不稳定的样本。在多个数学推理基准上的实验表明，SAGE显著加速收敛，性能超越静态基线，凸显了在推理对齐中，以策略感知、稳定性为导向的数据选择至关重要。
原文摘要 · Abstract (English)
Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving utility of training instances. This static approach often leads to inefficient or unstable optimization, as it wastes computation on trivial pairs with negligible gradients and suffers from noise induced by samples near uncertain decision boundaries. Facing these challenges, we propose SAGE (Stability-Aware Gradient Efficiency), a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates. Concretely, SAGE integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence with a fine-grained, stability-aware scoring function that prioritizes informative, confident errors while filtering out unstable samples. Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines, highlighting the critical role of policy-aware, stability-conscious data selection in reasoning alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。