arXiv:2608.19197cs.CLcs.AI2026-08被引 1

让大模型自己设计挑战任务,实现持续自我进化。

SPADE: Self-Play in Adaptive Synthetic Executable Environments

论文配图:SPADE: Self-Play in Adaptive Synthetic Executable Environments
图 1 · 摘自论文原文
  • 一个大模型同时扮演出题人和解题人,动态生成可执行的复杂任务环境。
  • 在8个基准上平均提升5.3分,工具使用任务提升最高达13.9分。
  • 适合研究开放世界智能体、自适应训练环境的开发者与研究人员。

持续自我改进需要不断生成多样化且自适应的目标。当前语言智能体的训练环境(人工构建、静态合成或固定验证)在模型规模扩大时仍保持目标分布不变。我们提出SPADE(自适应可执行环境中的自博弈),一种自博弈强化学习框架:单一大模型同时承担两个角色——环境设计者,以类似OpenAI Gym的reset()/step()接口生成完整的、长周期的可执行训练环境;以及推理智能体,在环境中学习行动。两者均为状态化、多轮交互的环境(含状态转移、奖励函数与验证代码),统一涵盖推理问题与多步工具使用。通过估计推理智能体有无特权提示时的收益差距来计算其遗憾值;环境设计者据此优化,将任务精准定位在智能体能力边缘但依然可行。实验表明关键组件包括基于大规模预训练语料采样文档进行引导,以及累积环境记忆。当模型扩展至300亿参数时,SPADE在八个数学、科学、代码与推理基准上平均优于最强固定环境基线5.3分,工具使用场景下在BFCL-v4多轮任务上提升5.7分,在ACEBench-Agent上提升13.9分;游戏设置下性能优势随模型规模增大而增长。通过将环境设计本身变为可学习模块,SPADE朝着开放式的自我改进迈出了实质一步。

原文摘要 · Abstract (English)

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

自博弈环境设计大模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。