解决多领域推理模型训练中的梯度冲突问题，提升通用推理能力。
Advancing General-Purpose Reasoning Models with Modular Gradient Surgery
- 在Transformer模块层面进行梯度手术，缓解跨领域干扰
- 在数学、对话、指令遵循任务上分别提升4.3和4.5分（相对增益11%-16.6%）
- 适用于长期训练，对Llama与Qwen等大模型均有效
强化学习（RL）在大型推理模型（LRMs）发展中起核心作用，显著提升了可验证和开放式的推理能力。然而，由于领域间差异显著，单一通用模型在多个领域联合训练仍面临挑战。通过对两种常用策略——顺序强化学习（Sequential RL）和混合强化学习（Mixed RL）的系统研究，我们发现两者在行为和梯度层面均存在严重的跨领域干扰，导致整体性能提升有限。为此，我们提出模块化梯度手术（MGS），在Transformer内部模块层面解决梯度冲突。在Llama和Qwen模型上应用MGS，在三个代表性领域（数学、通用对话、指令遵循）中分别实现平均4.3（16.6%）和4.5（11.1%）的提升。进一步分析表明，MGS在长时间训练下依然有效。本研究揭示了多领域强化学习中的干扰根源，并为训练通用型推理模型提供了有效方案。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has played a central role in recent advances in large reasoning models (LRMs), yielding strong gains in verifiable and open-ended reasoning. However, training a single general-purpose LRM across diverse domains remains challenging due to pronounced domain heterogeneity. Through a systematic study of two widely used strategies, Sequential RL and Mixed RL, we find that both incur substantial cross-domain interference at the behavioral and gradient levels, resulting in limited overall gains. To address these challenges, we introduce **M**odular **G**radient **S**urgery (**MGS**), which resolves gradient conflicts at the module level within the transformer. When applied to Llama and Qwen models, MGS achieves average improvements of 4.3 (16.6\%) and 4.5 (11.1\%) points, respectively, over standard multi-task RL across three representative domains (math, general chat, and instruction following). Further analysis demonstrates that MGS remains effective under prolonged training. Overall, our study clarifies the sources of interference in multi-domain RL and presents an effective solution for training general-purpose LRMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。