arXiv:2608.21871cs.CLcs.LG2026-08

让大模型自己设计难题并训练对手,实现无需人工数据的推理能力提升。

The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning

论文配图:The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning
图 1 · 摘自论文原文
  • 将自博弈建模为追逃游戏,让模型自主调节任务难度以保持挑战性。
  • 在三个环境上优于主流基线,统一模型在九个外部测试集上表现更强。
  • 适合研究零样本推理、自生成任务与强化学习协同的学者参考。

基于可验证奖励的强化学习已成为提升大语言模型推理能力的主流方法,但依赖大量人工标注的任务集。零数据自博弈虽减少依赖,但现有方法仅通过试探候选任务并事后拒绝来判断可学性,无法学习任务应置于环境难度轴上的位置,且仅给予求解者稀疏的最终奖励。本文将零数据自博弈重构为追逃博弈:在LURE中,一个大模型作为逃逸者,在各环境的难度轴上设置任务以避开追捕者(规划-执行者);追捕者通过可验证交互追踪逃逸者。逃逸者基于捕获前沿奖励进行训练,当求解者在约一半的采样中成功捕获时奖励最高,从而将‘勉强可捕’转化为可学习的位置策略。追捕者则获得捕获锚定的密集过程信用,其验证进度单调增长,并通过轮次锚定的KL散度与最终捕获联合归一化,确保双方演化稳定。在三个可验证推理环境和三种骨干模型家族中,LURE在统一/专用设置下均超越先进基线;统一模型在九个来自三个任务族的外部基准上,展现出优于所有训练基线的综合零样本外推准确率。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards has become the dominant recipe for improving large language model reasoning, yet it presumes large human-curated task collections. Zero-data self-play removes this dependency, but existing methods vet learnability only by probing candidates and rejecting post hoc, never learning where along an environment's difficulty axis to place a task, and credit the solver with sparse terminal rewards alone. We recast zero-data self-play as a pursuit-evasion game: in LURE, an LLM evader positions tasks along each environment's difficulty axis to stay one step ahead of a planner-executor pursuer that hunts it down through verifiable interaction. The evader is trained on a capture-frontier reward that peaks when the solver captures it on exactly half of its rollouts, turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band. The pursuer earns capture-anchored dense process credit, in which monotone verifier progress is group-normalized jointly with the terminal capture under a round-anchored KL that keeps the co-evolution stable. Across three verifiable reasoning environments and three backbone families, LURE outperforms advanced baselines under unified/specialist settings, while the unified model attains stronger aggregate OOD zero-shot accuracy than all trained baselines across nine held-out benchmarks from three task families.

自博弈零样本推理强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。