让大模型在多个任务中都稳定表现,避免某些任务被忽略。
Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
- 动态调整任务权重,优先提升最差任务的表现。
- 在3任务和9任务设置下,最差任务准确率提升16%-28%。
- 训练效率更高,只需一半步数就达50%的最差任务性能。
基于强化学习的后训练方法GRPO广泛用于提升大语言模型在单个推理任务上的表现。然而,实际部署需要模型在多样任务中保持可靠性能。直接扩展GRPO至多任务时,常出现部分任务主导优化而其他任务停滞的问题。此外,不同任务中提示产生零优势(即零梯度)的频率差异,进一步扭曲了其对优化信号的有效贡献。为此,我们提出新型多任务GRPO(MT-GRPO)算法:(i) 动态调整任务权重,显式优化最差任务表现并促进各任务均衡进步;(ii) 引入比例保持采样器,确保任务级策略梯度反映调整后的权重。在3任务与9任务设置下的实验表明,MT-GRPO在最差任务准确率上持续优于基线。具体而言,在最差任务性能上,相比标准GRPO提升16%-28%,相比DAPO提升6%,同时保持有竞争力的平均准确率。此外,在3任务设置下,MT-GRPO仅需50%的训练步数即可达到50%最差任务准确率,显著提升了跨任务可靠性能的训练效率。
原文摘要 · Abstract (English)
RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across diverse tasks. A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks dominating optimization while others stagnate. Moreover, tasks can vary widely in how frequently prompts yield zero advantages (and thus zero gradients), which further distorts their effective contribution to the optimization signal. To address these issues, we propose a novel Multi-Task GRPO (MT-GRPO) algorithm that (i) dynamically adapts task weights to explicitly optimize worst-task performance and promote balanced progress across tasks, and (ii) introduces a ratio-preserving sampler to ensure task-wise policy gradients reflect the adapted weights. Experiments on both 3-task and 9-task settings show that MT-GRPO consistently outperforms baselines in worst-task accuracy. In particular, MT-GRPO achieves 16-28% and 6% absolute improvement on worst-task performance over standard GRPO and DAPO, respectively, while maintaining competitive average accuracy. Moreover, MT-GRPO requires 50% fewer training steps to reach 50% worst-task accuracy in the 3-task setting, demonstrating substantially improved efficiency in achieving reliable performance across tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。