提出自适应优势函数FADE，解决强化学习中策略梯度不稳定与多样性下降问题。
Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL
- 将优势函数分解为符号与难度两个正交轴，揭示更新失衡机制
- 在7B和32B模型上分别提前20k和2k步达到峰值性能
- 自动调节梯度权重，适合需要高准确率与多样性的代码生成任务
强化学习后训练显著提升大语言模型的推理能力，但存在训练不稳定与多样性崩溃问题。优势函数通过重构训练目标、重加权推进学习的轨迹，实现简单且有效。然而方法众多，难以抉择。本文提出统一框架，将任意优势函数分解为沿符号轴与难度轴的正负梯度质量。符号轴上更新失衡会破坏熵或权重几何；难度轴上聚焦难题虽增强信号，但需更大样本量。二者在训练过程中动态变化：探索阶段偏好平衡与硬问题聚焦，利用阶段则偏好抑制与中等聚焦。据此设计自适应优势算法FADE（Focal Advantage with Dynamic Entropy），可实时读取训练动态并自动调度梯度权重。FADE在7B规模下比最优静态基线提前20,000步达到峰值pass@1，在32B规模下提前2,000步，且在LiveCodeBench与AIME数据集上均实现最佳准确率-多样性权衡。
原文摘要 · Abstract (English)
Reinforcement learning post-training dramatically improves LLM reasoning, but suffers from training instability and diversity collapse. Advantage functions offer an appealing fix: they reshape the training objective, reweight which rollouts drive learning, and are trivial to implement. Yet a proliferation of methods makes it unclear which advantage to use and when. We cut through the confusion with a unifying framework that decomposes any advantage into its positive and negative gradient mass along two orthogonal axes. On the sign axis, imbalanced updates collapse either entropy or weight geometry. On the difficulty axis, hard-problem focus sharpens signal but costs sample size. Both trade-offs shift during training: exploration favors balance and hard focus; exploitation favors suppression and medium focus. This motivates FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage that reads training dynamics to schedule the gradient weight automatically. FADE reaches peak pass@1 20k steps earlier than the best static baseline at the 7B scale and 2k steps earlier at the 32B , while achieving the best accuracy-diversity trade-off across all pass@k on LiveCodeBench and AIME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。