用大模型自动生成对抗性环境与策略,实现持续进化学习。
COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Game
- 将环境与策略设计建模为零和博弈,实现双向对抗演化。
- 通过纳什均衡生成元策略,防止遗忘旧任务并适应新挑战。
- 无需人工干预,可自动构建复杂度递增的学习课程,适合开放域学习研究者。
构建持续进化的智能体面临的核心挑战是训练环境通常静态或手动设计,限制了持续学习与分布外泛化能力。本文提出COvolve框架,利用大语言模型(LLMs)生成可执行的Python代码形式的环境与代理策略。将环境与策略设计者之间的互动建模为两玩家零和博弈,确保对抗性共演化:环境暴露策略弱点,策略随之适应。该过程催生自动化课程,使环境与策略共同向更高复杂度演进。为保障鲁棒性并防止遗忘,计算零和博弈的混合策略纳什均衡(MSNE),得到元策略。该元策略确保智能体在学习解决未见环境的同时,不遗忘已解决的任务。在城市驾驶、符号迷宫求解与几何导航任务上的实验表明,COvolve能生成渐进复杂的环境。结果展示了大模型驱动共演化在无预设任务分布、无需人工干预下的开放域学习潜力。
原文摘要 · Abstract (English)
A central challenge in building continually improving agents is that training environments are typically static or manually constructed. This restricts continual learning and generalization beyond the training distribution. We address this with COvolve, a co-evolutionary framework that leverages large language models (LLMs) to generate both environments and agent policies, expressed as executable Python code. We model the interaction between environment and policy designers as a two-player zero-sum game, ensuring adversarial co-evolution in which environments expose policy weaknesses and policies adapt in response. This process induces an automated curriculum in which environments and policies co-evolve toward increasing complexity. To guarantee robustness and prevent forgetting as the curriculum progresses, we compute the mixed-strategy Nash equilibrium (MSNE) of the zero-sum game, thereby yielding a meta-policy. This MSNE meta-policy ensures that the agent does not forget to solve previously seen environments while learning to solve previously unseen ones. Experiments in urban driving, symbolic maze-solving, and geometric navigation showcase that COvolve produces progressively more complex environments. Our results demonstrate the potential of LLM-driven co-evolution to achieve open-ended learning without predefined task distributions or manual intervention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。