让智能体在不重置环境的情况下持续自我改进,实现游戏通关
Continual Harness: Online Adaptation for Self-Improving Foundation Agents

- 通过在线迭代优化提示、子任务和记忆,实现无需重置的自适应
- 在《宝可梦红/绿》中减少按键成本,接近人工专家水平
- 支持智能体自主学习与反馈闭环,适合长时序决策场景
编码工具如Claude Code和OpenHands为大模型提供了工具、记忆与规划能力,但缺乏适用于具身智能体长期部分可观测决策的类似机制。我们首次报告了Gemini Plays Pokemon(GPP)实验:通过人机协同迭代优化,GPP成为首个在困难模式下完成《宝可梦蓝/黄》和《水晶》且未失败一战的AI系统。在最难关卡中,智能体借助长上下文记忆,自发产生自我改进信号,与人类协同优化并行。本研究提出Continual Harness,完全去除人为干预,构建一种无重置的自适应具身智能体框架,从基础环境接口出发,智能体在行动中持续优化自身提示、子代理、技能与记忆,并利用历史轨迹数据。相比需重置的提示优化方法,Continual Harness可在单次运行中在线适应。在《宝可梦红/绿》上,其从零开始显著降低按键成本,弥补大部分与人工专家调优框架之间的差距,且收益随能力提升而增长。此外,我们引入模型自身的在线过程-奖励协同学习闭环:开源智能体在不断优化的框架中执行,其轨迹由前沿教师模型重标注,并用于更新模型,从而在不重置环境的前提下持续达成游戏里程碑。
原文摘要 · Abstract (English)
Coding harnesses such as Claude Code and OpenHands wrap foundation models with tools, memory, and planning, but no equivalent exists for embodied agents' long-horizon partial-observability decision-making. We first report our Gemini Plays Pokemon (GPP) experiments. With iterative human-in-the-loop harness refinement, GPP became the first AI system to complete Pokemon Blue, Yellow Legacy on hard mode, and Crystal without a lost battle. In the hardest stages, the agent itself began iterating on its strategy through long-context memory, surfacing emergent self-improvement signals alongside human-in-the-loop refinement. Continual Harness removes the human fully from this loop: a reset-free self-improving harness for embodied agents that formalizes and automates what we observed. Starting from only a minimal environment interface, the agent alternates between acting and refining its own prompt, sub-agents, skills, and memory, drawing on any past trajectory data. Prompt-optimization methods require episode resets; Continual Harness adapts online within a single run. On Pokemon Red and Emerald across frontier models, Continual Harness starting from scratch substantially reduces button-press cost relative to the minimalist baseline and recovers a majority of the gap to a hand-engineered expert harness, with capability-dependent gains, despite starting from the same raw interface with no curated knowledge, no hand-crafted tools, and no domain scaffolding. We then close the loop with the model itself: an online process-reward co-learning loop, in which an open-source agent's rollouts through the refining harness are relabeled by a frontier teacher and used to update the model, drives sustained in-game milestone progress on Pokemon Red without resetting the environment between training iterations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。