通过跨步控制缓解多领域强化学习中的高阶干扰,提升模型性能。
One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

- 基于前一时刻的输出足迹,动态评估并修正令牌级反向风险。
- 在Qwen3-30B-A3B上实现0.4822的平均性能,提升5.7%。
- 无需显式高阶微分,适合大语言模型多任务训练场景。
多领域强化学习可拓展大语言模型的推理能力,但联合训练常导致各领域性能下降并引发优化不稳定。现有方法多从单步视角分析干扰,依赖一阶梯度对齐或曲率代理。我们发现,即使同一点梯度近乎正交,连续更新仍可能在输出空间部分抵消彼此影响。本文提出OSOL方法,通过相邻检查点的词元概率轨迹,直接捕捉输出空间中的局部二阶交互,无需重建同步曲率。每轮选定关注领域,利用前一检查点轨迹评估词元级反弹风险,并在标准GRPO更新中施加按风险排序、自适应缩放的修正项。分析表明该修正有效抑制了跨步输出回溯。受控实验进一步验证,跨步回溯与后续任务损伤关联更强,且前序轨迹比海森代理更准确预测反弹风险。在Qwen3-30B-A3B上,OSOL达到0.4822的域宏平均得分,优于最强基线5.7%,且无需显式高阶微分。
原文摘要 · Abstract (English)
Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。