模型自动生成任务并自我学习,无需任何外部数据即可实现顶尖推理能力。
Absolute Zero: Reinforced Self-play Reasoning with Zero Data

- 模型自主设计任务并验证答案,通过代码执行器获取可验证奖励。
- 在无外部数据情况下,代码与数学推理性能达到当前最优水平。
- 适用于不同规模模型,适合研究自进化AI系统的人参考。
基于可验证奖励的强化学习(RLVR)已展现出提升大语言模型推理能力的潜力,其直接从结果奖励中学习。现有零样本设置的RLVR方法虽避免了推理过程标注,但仍依赖人工整理的问答数据集进行训练。高质量人工数据稀缺,制约了长期可扩展性,且未来超智能系统若仅依赖人类任务,学习潜力有限。为此,我们提出新范式Absolute Zero,单个模型通过自主提出最大化自身学习进展的任务并解决它们,完全不依赖外部数据。在此范式下,我们引入绝对零号推理器(AZR),利用代码执行器验证所提任务与答案,作为统一的可验证奖励来源,驱动开放但有根基的学习。尽管完全无外部数据训练,AZR在代码与数学推理任务上仍达到当前最佳表现,超越依赖数万条人工标注示例的现有零样本模型。此外,我们证明了AZR在不同模型规模下有效,并兼容多种模型架构。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。