arXiv:2505.14652cs.CL2025-05NeurIPS被引 142

让大模型在多个领域都具备强推理能力,突破数学编码局限。

General-Reasoner: Advancing LLM Reasoning Across All Domains

  • 用网络爬虫构建跨学科可验证问题数据集
  • 用生成式模型替代规则验证,支持链式思考与上下文理解
  • 在12个领域基准上表现优异,数学推理也更强

强化学习近年在提升大语言模型推理能力方面展现巨大潜力,尤其是Deepseek-R1-Zero提出的「零样本」强化学习,使基础模型可直接进行强化学习训练,无需中间监督微调。然而,当前研究主要集中在数学和编程领域,因数据丰富且答案易验证。这限制了模型在更广泛领域的适用性和泛化能力,因为其他领域问题答案形式多样、数据稀缺。本文提出General-Reasoner,一种新型训练范式,旨在提升大模型在多样化领域中的推理能力。核心贡献包括:(1) 构建大规模高质量跨学科问题数据集,通过网络爬取获得,覆盖物理、化学、金融、电子等多个领域;(2) 开发基于生成式模型的答案验证器,以链式思维和上下文感知能力替代传统规则验证。我们在涵盖12个不同领域基准(如MMLU-Pro、GPQA、SuperGPQA、TheoremQA、BBEH和MATH AMC)的测试中评估了一系列模型,结果表明General-Reasoner显著优于现有基线方法,在保持数学推理优势的同时,实现了稳健且通用的推理性能。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs). Particularly, the "Zero" reinforcement learning introduced by Deepseek-R1-Zero, enables direct RL training of base LLMs without relying on an intermediate supervised fine-tuning stage. Despite these advancements, current works for LLM reasoning mainly focus on mathematical and coding domains, largely due to data abundance and the ease of answer verification. This limits the applicability and generalization of such models to broader domains, where questions often have diverse answer representations, and data is more scarce. In this paper, we propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing a generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of-thought and context-awareness. We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 benchmarks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that General-Reasoner outperforms existing baseline methods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.

大模型推理跨领域强化学习生成验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。