用强化学习研究多领域推理,发现不同任务间既有互助也有冲突。
Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
- 在数学、编程、逻辑三领域上测试单任务与多任务训练效果
- 发现跨领域训练时存在相互促进或干扰的复杂关系
- 适合关注大模型综合推理能力提升的研究者阅读
强化学习结合可验证奖励(RLVR)已成为提升大模型推理能力的有效范式。现有研究多聚焦于单一推理领域,如数学解题、代码生成或逻辑推理,但现实场景中需综合运用多种认知技能。然而,这些推理能力在强化学习下的交互机制仍不清晰。为此,本文在RLVR框架下系统研究数学推理、代码生成和逻辑谜题求解三个核心领域的多领域推理。通过四项关键实验:(1)基于GRPO算法与Qwen-2.5-7B模型族,评估单领域数据训练下的领域内提升与跨域泛化能力;(2)分析多任务联合训练中的相互促进与冲突现象;(3)比较基础模型与指令微调模型在相同强化学习配置下的表现差异;(4)系统探究课程学习策略、奖励设计变化及语言特性对训练的影响。大量实验证明,领域间存在复杂的动态交互关系,揭示了影响专业化与通用化推理表现的关键因素,为优化大模型多领域推理的强化学习方法提供重要指导。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of LLMs. Existing research has predominantly concentrated on isolated reasoning domains such as mathematical problem-solving, coding tasks, or logical reasoning. However, real world reasoning scenarios inherently demand an integrated application of multiple cognitive skills. Despite this, the interplay among these reasoning skills under reinforcement learning remains poorly understood. To bridge this gap, we present a systematic investigation of multi-domain reasoning within the RLVR framework, explicitly focusing on three primary domains: mathematical reasoning, code generation, and logical puzzle solving. We conduct a comprehensive study comprising four key components: (1) Leveraging the GRPO algorithm and the Qwen-2.5-7B model family, our study thoroughly evaluates the models' in-domain improvements and cross-domain generalization capabilities when trained on single-domain datasets. (2) Additionally, we examine the intricate interactions including mutual enhancements and conflicts that emerge during combined cross-domain training. (3) To further understand the influence of SFT on RL, we also analyze and compare performance differences between base and instruct models under identical RL configurations. (4) Furthermore, we delve into critical RL training details, systematically exploring the impacts of curriculum learning strategies, variations in reward design, and language-specific factors. Through extensive experiments, our results offer significant insights into the dynamics governing domain interactions, revealing key factors influencing both specialized and generalizable reasoning performance. These findings provide valuable guidance for optimizing RL methodologies to foster comprehensive, multi-domain reasoning capabilities in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。