用分位数替代均值基线，稳定大模型强化学习推理中的熵震荡问题。
Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
- 改用分位数基线，按难度分组动态调节奖励信号。
- 实验显示熵稳定，80%响应优势为零，推理准确率持续提升。
- 适合追求稳定强化学习训练的大模型研究者使用。
基于可验证奖励的强化学习（RLVR）能增强大模型的推理能力，但训练常在熵坍缩与熵爆炸间震荡。我们发现根源在于价值无关强化学习（如GRPO和DAPO）中使用的均值基线，在奖励异常值下错误惩罚负优势样本。为此提出分位数优势估计（QAE），用组内分位数基线替代均值。QAE实现响应级两阶段调控：对难题（p ≤ 1−K）强化稀有成功，对易题（p > 1−K）聚焦剩余失败。在一阶Softmax更新下，证明了双侧熵安全性，即一步熵变化有上下界，有效抑制爆炸并防止坍缩。实验证明，此微小改动使熵保持稳定，信用分配更稀疏（调参后约80%响应优势为零），在Qwen3-8B/14B-Base上持续提升AIME 2024/2025与AMC 2023的pass@1性能。结果表明，基线设计而非词元级启发式，是扩展RLVR的关键机制。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between {entropy collapse} and {entropy explosion}. We trace both hazards to the mean baseline used in value-free RL (e.g., GRPO and DAPO), which improperly penalizes negative-advantage samples under reward outliers. We propose {Quantile Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: on hard queries (p <= 1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it targets remaining failures. Under first-order softmax updates, we prove {two-sided entropy safety}, giving lower and upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results identify {baseline design} -- rather than token-level heuristics -- as the primary mechanism for scaling RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。