用群体自对弈让大模型互相出题解题,显著提升推理能力。
PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play
- 师生群体分工:教师出题,学生解题,程序化验证奖励
- 群体持续进化,问题难度递增,解题率波动但覆盖更广
- 70亿参数下秒级更新权重,弱成员也超越单模型基线
我们提出 PopuLoRA,一种基于种群的非对称自对弈框架,用于大语言模型在强化学习中使用可验证奖励(RLVR)进行后训练。教师和学生均为共享冻结基础模型上的专用 LoRA 适配器:教师生成问题,学生在程序化验证器下求解,子群体间交叉评估替代了单智能体自校准的局限。一组 LoRA 权重空间演化算子(突变与交叉,可在数秒内生成同秩种群成员)取代了 70 亿参数规模下的种群训练循环中的替换步骤。我们在 Absolute Zero Reasoner 上实例化 PopuLoRA,与计算量匹配的单代理基线对比。单代理自校准为能可靠求解的简单问题,而群体进入协同进化竞赛:教师提出越来越复杂的问题,学生求解率震荡,问题空间覆盖率持续扩展。尽管训练期间平均奖励较低,群体均值在三个代码基准(HumanEval+、MBPP+、LiveCodeBench)和七个数学基准(AIME 24/25、AMC 23、MATH-500、Minerva、GSM8K、OlympiadBench)上均优于基线,且群体中最弱成员整体也超过基线。
原文摘要 · Abstract (English)
We introduce PopuLoRA, a population-based asymmetric self-play framework for reinforcement learning with verifiable rewards (RLVR) post-training of LLMs. Teachers and students are specialised LoRA adapters on a shared frozen base: teachers propose problems, matched students solve them under a programmatic verifier, and cross-evaluation between sub-populations replaces the self-calibration that limits single-agent self-play. A family of LoRA weight-space evolution operators (mutations and crossovers that produce same-rank population members in seconds) serves as the replacement step of a population-based training loop at 7B scale. We instantiate PopuLoRA on top of Absolute Zero Reasoner and compare it against a per-adapter compute-matched single-agent baseline. Where the single agent self-calibrates to generating easy problems it can reliably solve, the population enters a co-evolutionary arms race: teachers produce increasingly complex problems, student solve rates oscillate, and problem-space coverage keeps expanding throughout training. Despite lower training-time reward, the population mean outperforms the baseline on three code benchmarks (HumanEval+, MBPP+, LiveCodeBench) and seven math benchmarks (AIME 24/25, AMC 23, MATH-500, Minerva, GSM8K, OlympiadBench), and even the weakest member of the population beats the baseline on aggregate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。