动态调整训练分布,让大模型更专注攻克难题。
Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning
- 用在线难度分类器实时分组提示,动态调整训练重点。
- 在固定算力下,推理准确率提升10.6%和10.1%。
- 适合追求长尾问题性能的LLM训练者使用。
大语言模型(LLM)推理能力的提升越来越依赖后训练损失函数与对齐策略的优化。然而,标准强化学习范式如分组相对策略优化(GRPO)受限于静态均匀性:提示采样均匀、每提示滚动次数固定。面对异构且重尾的推理数据,这种结构导致算力浪费在已解决模式上,而难以覆盖长尾难题。为此,我们提出多对抗分组分布鲁棒优化(GDRO),一种以优化为核心的框架,通过动态调整训练分布超越均匀模型。引入在线难度分类器,将提示划分为动态的pass@k难度组。提出两种独立的GDRO机制:(1)提示-GDRO,采用EMA去偏乘法权重贝叶斯采样器,聚焦高难度边界,无频率偏差地加权持续困难组;(2)滚动-GDRO,使用影子价格控制器,在固定平均预算下重新分配滚动次数,最大化硬任务的梯度方差降低(算力中立)。为两者提供无遗憾保证,并给出方差代理分析,支持平方根最优滚动分配。在包含14.1k样本的DAPO数据集上,使用Qwen3-Base模型验证,提示-GDRO与滚动-GDRO分别在1.7B、4B、8B规模下实现相对于GRPO基准的+10.6%与+10.1%的平均相对提升(pass@8准确率)。定性分析显示出现自适应课程:对抗机制持续向演进中的推理前沿倾斜资源,显著提升模型表现。
原文摘要 · Abstract (English)
Recent progress in Large Language Model (LLM) reasoning is increasingly driven by the refinement of post-training loss functions and alignment strategies. However, standard Reinforcement Learning (RL) paradigms like Group Relative Policy Optimization (GRPO) remain constrained by static uniformity: uniform prompt sampling and a fixed number of rollouts per prompt. For heterogeneous, heavy-tailed reasoning data, this creates structural inefficiencies that waste compute on already-solved patterns while under-training the long tail of hard problems. To address this, we propose Multi-Adversary Group Distributionally Robust Optimization (GDRO), an optimization-first framework that moves beyond uniform reasoning models by dynamically adapting the training distribution. We introduce an Online Difficulty Classifier that partitions prompts into dynamic pass@k difficulty groups. We then propose two independent GDRO games for post-training: (1) Prompt-GDRO, which employs an EMA-debiased multiplicative-weights bandit sampler to target the intensive difficulty margin and upweight persistently hard groups without frequency bias; and (2) Rollout-GDRO, which uses a shadow-price controller to reallocate rollouts across groups, maximizing gradient variance reduction on hard tasks under a fixed mean budget (compute-neutral). We provide no-regret guarantees for both controllers and additionally a variance-proxy analysis motivating a square-root optimal rollout allocation for Rollout-GDRO. We validate our framework on the DAPO 14.1k dataset using Qwen3-Base models. Prompt-GDRO and Rollout-GDRO achieve average relative gains of +10.6% and +10.1%, respectively, in pass@8 accuracy across 1.7B, 4B, and 8B scales compared to the GRPO baseline. Qualitative analysis shows an emergent curriculum: the adversaries shift resources to the evolving reasoning frontier, enhancing the reasoning model's performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。