用大模型知识直接生成编程化技能，让强化学习更快上手。
InnateCoder: Learning Programmatic Options with Foundation Models
- 从大模型中零样本提取人类编程经验，生成可复用的长期动作
- 在MicroRTS和Karel机器人任务中，采样效率提升显著
- 适合想快速训练程序化策略的研究者和开发者
在非迁移学习场景下，强化学习智能体需从零开始学习基础技能，过程缓慢。本文提出InnateCoder系统，利用大模型中编码的人类知识，生成以程序化形式存在的“先天技能”（即时间扩展动作或选项）。与现有方法不同，InnateCoder在零样本设置下从大模型中学习这些选项，而非通过环境交互积累经验。随后，系统通过组合这些选项程序构建更复杂的大规模程序策略。我们假设该方法能提升程序化策略学习的采样效率。在MicroRTS和Karel the Robot上的实验证明，使用选项的InnateCoder相比不使用选项或基于经验学习选项的版本，采样效率更高。
原文摘要 · Abstract (English)
Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder, a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode "innate skills" in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder's way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。