跨领域重构强化学习提升大模型推理能力，发现训练需匹配预训练内容。
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
- 构建6领域9.2万可验证推理数据集，支持跨域强化学习训练
- 跨域训练在数学代码等常见领域效果好，冷门领域需本域训练才有效
- 发布Guru-7B/32B模型，开源全部代码与数据，适合通用推理研究者
强化学习（RL）已成为提升大语言模型（LLM）推理能力的有前景方法，但现有工作多集中于数学和代码领域，限制了对更广泛推理应用的理解。关键挑战在于缺乏跨多样化推理领域的可靠、可扩展的奖励信号。本文提出Guru，一个包含92,000个可验证示例的精选强化学习推理语料库，覆盖数学、代码、科学、逻辑、模拟和表格六大领域，每个领域均通过领域特定奖励设计、去重与过滤确保训练可靠性与有效性。基于Guru，我们系统重审了主流的LLM推理强化学习结论，发现不同领域表现差异显著：以往认为RL仅激发预训练模型已有知识，而本研究揭示更复杂模式——在预训练中频繁出现的领域（数学、代码、科学）可通过跨域训练获益，而在预训练中曝光较少的领域（逻辑、模拟、表格）则需本域训练才能实现显著性能提升，表明强化学习可能促进真实技能习得。最后，我们推出Guru-7B和Guru-32B两个模型，在涵盖六个领域的17项任务评估中，分别超越最佳基线7.9%和6.7%，达到当前公开数据训练模型的最先进水平。同时，我们的模型显著提升了基础模型在复杂任务上的Pass@k表现，尤其针对预训练中罕见的任务类型。数据、模型、训练与评估代码已开源，地址：https://github.com/LLM360/Reasoning360。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a promising approach to improve large language model (LLM) reasoning, yet most open efforts focus narrowly on math and code, limiting our understanding of its broader applicability to general reasoning. A key challenge lies in the lack of reliable, scalable RL reward signals across diverse reasoning domains. We introduce Guru, a curated RL reasoning corpus of 92K verifiable examples spanning six reasoning domains--Math, Code, Science, Logic, Simulation, and Tabular--each built through domain-specific reward design, deduplication, and filtering to ensure reliability and effectiveness for RL training. Based on Guru, we systematically revisit established findings in RL for LLM reasoning and observe significant variation across domains. For example, while prior work suggests that RL primarily elicits existing knowledge from pretrained models, our results reveal a more nuanced pattern: domains frequently seen during pretraining (Math, Code, Science) easily benefit from cross-domain RL training, while domains with limited pretraining exposure (Logic, Simulation, and Tabular) require in-domain training to achieve meaningful performance gains, suggesting that RL is likely to facilitate genuine skill acquisition. Finally, we present Guru-7B and Guru-32B, two models that achieve state-of-the-art performance among open models RL-trained with publicly available data, outperforming best baselines by 7.9% and 6.7% on our 17-task evaluation suite across six reasoning domains. We also show that our models effectively improve the Pass@k performance of their base models, particularly on complex tasks less likely to appear in pretraining data. We release data, models, training and evaluation code to facilitate general-purpose reasoning at: https://github.com/LLM360/Reasoning360
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。