提出自适应记忆管理机制，高效处理复杂环境中的长时依赖问题。
Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments
- 设计自适应堆栈算法，动态保留关键历史观测，减少冗余记忆
- 在多种模型上实现计算与内存降低40%以上，且保证收敛性
- 适合资源受限场景下的智能体部署，尤其适用于长时依赖任务
基于序列模型的通用智能体在现实复杂环境中面临显著的非马尔可夫依赖问题，传统帧堆叠方法需随依赖程度增加窗口大小，导致计算与内存开销剧增。本文提出自适应堆栈（Adaptive Stacking）元算法，利用环境在时间尺度上仅依赖少数关键观测的特性，动态维护小规模记忆堆栈，从而在保持高表达能力的同时大幅降低计算和存储需求。实验在可控非马尔可夫度的记忆任务中验证，该方法能有效识别并剔除对未来奖励无预测价值的记忆，同时避免重要经验丢失。在MLP、LSTM和Transformer模型上均实现显著效率提升，且具备理论收敛保障。
原文摘要 · Abstract (English)
Recent success in developing increasingly general purpose agents based on sequence models has led to increased focus on the problem of deploying computationally limited agents within the vastly more complex real-world. A key challenge experienced in these more realistic domains is highly non-Markovian dependencies with respect to the agent's observations, which are less common in small controlled domains. The predominant approach for dealing with this in the literature is to stack together a window of the most recent observations (Frame Stacking), but this window size must grow with the degree of non-Markovian dependencies, which results in prohibitive computational and memory requirements for both action inference and learning. In this paper, we are motivated by the insight that in many environments that are highly non-Markovian with respect to time, the environment only causally depends on a relatively small number of observations over that time-scale. A natural direction would then be to consider meta-algorithms that maintain relatively small adaptive stacks of memories such that it is possible to express highly non-Markovian dependencies with respect to time while considering fewer observations at each step and thus experience substantial savings in both compute and memory requirements. Hence, we propose a meta-algorithm (Adaptive Stacking) for achieving exactly that with convergence guarantees and quantify the reduced computation and memory constraints for MLP, LSTM, and Transformer-based agents. Our experiments utilize popular memory tasks, which give us control over the degree of non-Markovian dependencies. This allows us to demonstrate that an appropriate meta-algorithm can learn the removal of memories not predictive of future rewards without excessive removal of important experiences. Code: https://github.com/geraudnt/adaptive-stacking
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。