用外部记忆检索关键经验，让强化学习在少样本下更快适应新任务。
Retrieval-Augmented Decision Transformer: External Memory for In-context RL
- 通过检索机制从外部记忆中提取相关子轨迹，减少上下文依赖。
- 在网格世界中仅用少量上下文就超越基线模型性能。
- 无需训练的通用检索模块，适合复杂环境中的快速决策场景。
上下文学习（ICL）是指模型通过观察上下文中的少量示例来学习新任务的能力。尽管该能力在自然语言处理中广泛存在，近期也被发现于强化学习（RL）设置中。然而，先前的上下文强化学习方法需要将完整轨迹放入智能体的上下文，而复杂环境中通常存在长轨迹与稀疏奖励，导致这些方法仅适用于短轨迹的简单环境。为此，我们提出检索增强型决策变换器（RA-DT），采用外部记忆机制存储过往经验，并仅检索当前情境相关的子轨迹。该检索组件无需训练，可完全领域无关。我们在网格世界、机器人模拟及程序生成视频游戏上评估了RA-DT的表现。在网格世界中，RA-DT以远低于基线的上下文长度实现了更优性能。此外，我们揭示了现有上下文强化学习方法在复杂环境中的局限性，并讨论了未来方向。为促进后续研究，我们公开了四个环境的数据集。
原文摘要 · Abstract (English)
In-context learning (ICL) is the ability of a model to learn a new task by observing a few exemplars in its context. While prevalent in NLP, this capability has recently also been observed in Reinforcement Learning (RL) settings. Prior in-context RL methods, however, require entire episodes in the agent's context. Given that complex environments typically lead to long episodes with sparse rewards, these methods are constrained to simple environments with short episodes. To address these challenges, we introduce Retrieval-Augmented Decision Transformer (RA-DT). RA-DT employs an external memory mechanism to store past experiences from which it retrieves only sub-trajectories relevant for the current situation. The retrieval component in RA-DT does not require training and can be entirely domain-agnostic. We evaluate the capabilities of RA-DT on grid-world environments, robotics simulations, and procedurally-generated video games. On grid-worlds, RA-DT outperforms baselines, while using only a fraction of their context length. Furthermore, we illuminate the limitations of current in-context RL methods on complex environments and discuss future directions. To facilitate future research, we release datasets for four of the considered environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。