让Transformer在上下文中同时学环境模型和优化策略，少交互也能超好表现。
Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning
- 上下文内联合学习环境动态与策略，无需额外训练。
- 在多种环境上用更少交互达到顶尖性能。
- 适合想提升样本效率的强化学习研究者。
近期研究表明，Transformer可通过模仿现有强化学习算法实现上下文强化学习，从而在不更新参数的情况下高效适应未见过的任务。然而，这些模型也继承了所模仿算法的次优行为，主要源于其渐进式更新机制。模型基规划为解决此问题提供新思路，使模型能在行动前模拟潜在结果，从而突破次优行为限制。本文提出上下文内规划蒸馏（DICP），一种Transformer在上下文中同时学习环境动态与优化策略的模型基强化学习框架。我们在包括Darkroom变体和Meta-World在内的多种离散与连续环境中评估DICP，结果表明，其性能达当前最优，且所需环境交互次数显著少于基线方法，涵盖模型无关方法及现有元强化学习方法。
原文摘要 · Abstract (English)
Recent studies have shown that Transformers can perform in-context reinforcement learning (RL) by imitating existing RL algorithms, enabling sample-efficient adaptation to unseen tasks without parameter updates. However, these models also inherit the suboptimal behaviors of the RL algorithms they imitate. This issue primarily arises due to the gradual update rule employed by those algorithms. Model-based planning offers a promising solution to this limitation by allowing the models to simulate potential outcomes before taking action, providing an additional mechanism to deviate from the suboptimal behavior. Rather than learning a separate dynamics model, we propose Distillation for In-Context Planning (DICP), an in-context model-based RL framework where Transformers simultaneously learn environment dynamics and improve policy in-context. We evaluate DICP across a range of discrete and continuous environments, including Darkroom variants and Meta-World. Our results show that DICP achieves state-of-the-art performance while requiring significantly fewer environment interactions than baselines, which include both model-free counterparts and existing meta-RL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。