对比SFT与RL在多任务训练中的差异，发现RL更稳定且可并行。
SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
- RL通过稀疏正交更新实现多任务共存，减少冲突。
- 实验显示RL梯度方差小，优化方向近乎正交，提升稳定性。
- 提出Parallel-RL框架，支持任务解耦，效率更高。
监督微调（SFT）与强化学习（RL）在提升大语言模型多任务推理能力时表现出根本性差异。初步实验发现：在多阶段训练中，SFT存在严重任务冲突，而RL则能实现不同任务的稳定共存。实证分析表明，该现象源于参数层面——RL促使跨任务更新趋于稀疏且近似正交。我们通过多任务梯度干扰分析提供了理论解释：SFT的干扰受梯度范数限制，随梯度绝对值增长；而RL的干扰受方差限制，由优势归一化和在线策略优化带来的梯度方差所约束。这一小方差边界导致任务间优化方向近乎正交。基于此，我们提出Parallel-RL，一种解耦多任务训练的新范式，显著提升训练效率与灵活性。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。