动态调整模型微调与强化学习的混合比例，提升训练效率。
GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
- 根据梯度方差和信号差异在线计算混合权重
- 在多个任务上显著优于固定混合策略，大模型收益更明显
- 几乎不增加训练开销，适合大规模模型优化
混合后训练通常结合监督微调与强化学习，但固定混合策略无法适应两类信号噪声随时间变化的情况。本文提出GAC，一种噪声感知控制器，通过在线估计梯度方差和两种训练信号间的分歧，自适应地生成混合权重。该方法引入平滑、先验引导和有界更新机制，同时复用现有训练张量。在数学、代码、科学和逻辑等基准测试中，GAC持续优于强固定的及规则基线，尤其在大模型上表现更优，且训练开销低于1%。
原文摘要 · Abstract (English)
Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。