为强化学习智能体记忆能力提供分类与评估标准
Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation
- 基于认知科学定义记忆类型，区分长短时、陈述性与程序性记忆
- 提出统一实验方法，避免因评估不当导致误判
- 适合研究记忆机制或对比智能体性能的RL学者参考
将记忆引入强化学习（RL）智能体对众多任务至关重要，尤其在依赖历史信息、适应新环境和提升样本效率方面。然而，'记忆'概念涵盖广泛，且缺乏统一验证方法，导致对智能体记忆能力的判断错误，阻碍客观比较。本文通过借鉴认知科学，提出长时与短时、陈述性与程序性记忆的明确界定，对智能体记忆进行分类，构建稳健的评估实验方法并实现标准化。通过在不同RL智能体上实证，证明遵循该方法的重要性；反之则可能导致误判。该框架有助于更准确地分析和比较各类记忆增强型智能体的性能。
原文摘要 · Abstract (English)
The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the use of past information, adaptation to novel environments, and improved sample efficiency. However, the term "memory" encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term vs. short-term memory and declarative vs. procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。