提出自适应课程机制,让多领域推理模型更高效地迁移知识。
Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

- 基于梯度几何对齐估算跨领域迁移能力,动态调整训练顺序。
- 在六领域任务中,相比基准方法最高提升2.8分(10%相对)。
- 适合需要多领域通用推理能力的研究者和开发者。
强化学习结合可验证奖励(RLVR)已从单领域扩展至涵盖数学、编程与科学的多领域推理任务。然而,训练课程(各领域采样频率)通常固定或人工设定,而不同领域间推理能力的迁移效果不均。现有基于可学习性的课程虽能响应当前进展,却忽略某领域更新是否对其他领域有益。本文提出迁移感知课程(TAC),一种类老虎机的在线课程策略,优先选择其更新能广泛惠及其余领域的任务。TAC复用训练过程中的已有信号:各领域优势反映局部可学习性,而来自GRPO步骤的投影梯度通过梯度几何对齐估计跨域迁移性,开销极低(<1%实际运行时间)。在六领域推理套件上,TAC在Qwen3-1.7B和Llama3.2-3B上均取得最佳宏观平均准确率,优于比例随机采样、人工设计调度及仅基于可学习性的带状算法,且相较后者最高提升2.8点(10%相对)。消融实验显示移除迁移性项后性能急剧下降,且在不平衡数据混合下,TAC仍保持稳健,而仅依赖可学习性的课程会过度集中于主导领域。结果表明,跨领域迁移性是多领域RLVR课程设计的关键信号。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。