无需外部数据,通过教练-玩家协作提升模型推理能力
CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning
- 教练提出任务,玩家执行,双方共同优化推理能力
- 在数学推理上平均提升4.9分,分布外任务提升5.4分
- 适合无标注数据场景下的强化学习模型训练
大型语言模型在复杂推理中展现潜力,但其发展受限于对大量高质量人工标注任务和标签的依赖,无论是通过监督微调还是特定数据上的强化学习。这种依赖使监督训练模式日益不可持续,实际中已显现出可扩展性下降的迹象。为突破此限制,我们提出CPMobius,一种面向推理模型的数据自由强化学习协作范式。不同于传统对抗自对弈,CPMobius受现实世界体育合作与多智能体协作启发,将教练与玩家视为独立但协作的角色。教练针对玩家能力设计指令,并根据玩家性能变化获得奖励;玩家则因解决教练生成的更具挑战性的任务而获奖励。这一协作优化循环旨在直接提升玩家的数学推理能力。显著的是,该方法无需任何外部训练数据即可实现显著提升,在Qwen2.5-Math-7B-Instruct上整体平均准确率提高4.9,分布外平均提高5.4,整体准确率超越RENT 1.5,分布外准确率超越R-zero 4.2。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated strong potential in complex reasoning, yet their progress remains fundamentally constrained by reliance on massive high-quality human-curated tasks and labels, either through supervised fine-tuning (SFT) or reinforcement learning (RL) on reasoning-specific data. This dependence renders supervision-heavy training paradigms increasingly unsustainable, with signs of diminishing scalability already evident in practice. To overcome this limitation, we introduce CPMöbius (CPMobius), a collaborative Coach-Player paradigm for data-free reinforcement learning of reasoning models. Unlike traditional adversarial self-play, CPMöbius, inspired by real world human sports collaboration and multi-agent collaboration, treats the Coach and Player as independent but cooperative roles. The Coach proposes instructions targeted at the Player's capability and receives rewards based on changes in the Player's performance, while the Player is rewarded for solving the increasingly instructive tasks generated by the Coach. This cooperative optimization loop is designed to directly enhance the Player's mathematical reasoning ability. Remarkably, CPMöbius achieves substantial improvement without relying on any external training data, outperforming existing unsupervised approaches. For example, on Qwen2.5-Math-7B-Instruct, our method improves accuracy by an overall average of +4.9 and an out-of-distribution average of +5.4, exceeding RENT by +1.5 on overall accuracy and R-zero by +4.2 on OOD accuracy. Our codebase has been released at https://github.com/thunlp/CPMobius.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。