模型自动生成训练任务,零数据实现推理能力持续进化
R-Zero: Self-Evolving Reasoning LLM from Zero Data

- 用两个角色模型互演:挑战者出难题,求解者破题,共同进化
- 在数学和通用推理上分别提升6.49和7.54分,无需人工标注
- 适合研究自主智能、自进化AI的学者与开发者
自进化大语言模型通过自主生成、优化和学习自身经验,为通向超智能提供可扩展路径。然而,现有方法仍严重依赖大量人工标注的任务与标签,通常通过微调或强化学习实现,成为突破人类智能上限的根本瓶颈。为此,我们提出R-Zero,一个完全自主的框架,从零开始生成训练数据。基于单一基础模型,R-Zero初始化两个独立角色模型:挑战者与求解者。二者分别优化并协同进化:挑战者因提出逼近求解者能力边界的任务而获奖励,求解者则因解决由挑战者提出的越来越难的问题而获奖励。该过程生成针对性强、自我提升的课程,无需任何预设任务与标签。实验表明,R-Zero显著提升不同主干模型的推理能力,例如使Qwen3-4B-Base在数学推理基准上提升+6.49,在通用领域推理基准上提升+7.54。
原文摘要 · Abstract (English)
Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。