让智能体自动发现并进化技能程序,无需人工设计奖励。
CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs
- 用可执行的代码程序表示技能,形成递归奖励链。
- 在Craftax上性能比之前方法高6倍,能自动生成铁器和挖钻石。
- 完全自主训练,适合研究通用智能体与长程任务求解。
通用智能的核心能力是能够自主地持续拓展和演化其掌握的技能。尽管近期基于基础模型(FM)的方法在这一方向展现了潜力,但通常仍依赖大量人工干预,限制了其在新环境中的迁移能力。为此,我们提出连续开放式的技能发现与演化框架——CODE-SHARP,该框架利用基础模型从零开始自动构建一个包含Python代码的技能档案,通过强化学习直接从源码训练通用智能体策略。这些代码程序称为技能的分层奖励程序(SHARPs),每个程序定义局部成功条件及前置条件,由先前发现的SHARPs实现。运行时,根据当前状态动态调度前置链路,沿途奖励完成节点,使智能体只需学习新增的边际行为,从而高效习得长时序技能,且无需预设奖励。在Craftax-Classic和XLand上,由CODE-SHARP完全自主训练的智能体分别取得6倍和2.6倍于以往工作的中位性能提升,且唯一能实现铁具制作与钻石开采。扩展至Craftax-Extended后,系统训练出涵盖90多个已发现SHARPs的通用智能体,可在零样本情况下解决复杂长时序任务,表现媲美基于真实奖励训练的模型。
原文摘要 · Abstract (English)
A core quality of general intelligence is the ability to open-endedly expand and evolve its set of mastered skills autonomously. While recent Foundation Model (FM) driven approaches have shown promising results towards this goal, they typically rely on significant human-in-the-loop engineering, limiting their transferability to novel environments. To address this, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a framework that leverages FMs to open-endedly grow and evolve an archive of Python programs encoding skills to train a generalist agent policy entirely from scratch via reinforcement learning, directly from source code. These programs, termed Skills as Hierarchical Reward Programs (SHARPs), each encode a local success condition and a set of prerequisites delegated to previously discovered SHARPs. At runtime, SHARPs dynamically route the agent through their prerequisite chain based on the current state, rewarding each completion along the way, requiring the agent to learn only the marginal behaviour each new SHARP introduces, enabling efficient learning of long-horizon skills without any pre-defined rewards. On Craftax-Classic and XLand, agents trained fully autonomously by CODE-SHARP outperform previous works by 6x and 2.6x in median performance and are the only agents capable of crafting iron tools and mining diamonds. Scaled to Craftax-Extended, CODE-SHARP trains a generalist agent on over 90 discovered SHARPs, enabling the agent to solve challenging long-horizon tasks zero-shot, matching agents trained on ground-truth rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。