通过新环境研究强化学习中记忆的作用机制。
Investigating Memory in Model-Free RL with POPGym Arcade
- 设计可对比全观与部分观测的仿真环境，分析策略如何依赖记忆
- 发现价值函数会错误地将奖励归因于无关历史，导致记忆污染
- 适合研究记忆机制、鲁棒性或对抗攻击的RL研究人员
如何分析深度强化学习中的记忆？我们引入工具来分析部分可观测环境下策略的行为，揭示智能体如何利用记忆做决策。为此，我们提出POPGym Arcade——一系列受Atari启发、支持硬件加速的环境，共享统一的观测与动作空间。每个环境提供完全可观测和部分可观测两种变体，支持对可观测性影响的反事实研究。我们发现，公平比较必须依赖受控实验，并识别出一种病理：价值函数会将信用错误地扩散到无关的历史上。利用这一现象，我们证明了分布外场景可长期干扰记忆，显著改变未来策略行为。代码已开源。
原文摘要 · Abstract (English)
How should we analyze memory in deep RL? We introduce tools for analyzing policies under partial observability and revealing how agents use memory to make decisions. To utilize these tools, we present POPGym Arcade, a collection of Atari-inspired, hardware-accelerated environments sharing a single observation and action space. Each environment provides fully and partially observable variants, enabling counterfactual studies on observability. We find that controlled studies are necessary for fair comparisons and identify a pathology where value functions smear credit over irrelevant history. Using this pathology, we demonstrate how out-of-distribution scenarios can contaminate memory, perturbing the policy far into the future. Our code is available at https://github.com/bolt-research/popgym-arcade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。