为应对环境变化，提出一种基于空间采样的值衰减遗忘机制。
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
- 通过空间采样设计值函数衰减，实现无监督环境漂移适应。
- 在DQN和SAC上验证，显著提升非平稳环境下的长期回报。
- 适合研究动态环境中的强化学习鲁棒性问题。
啮齿类动物（如小鼠）在环境参数变化（漂移）且无明确变化信号时仍能调整行为，这种能力可通过遗忘机制建模。非平稳强化学习（NSRL）旨在使先进RL方法适应动态环境，但通常需依赖部分精确信息（如任务ID或上下文）。本文提出一种名为「空间采样值衰减」（Space-sampled Value Decay）的显式遗忘机制，用于基于值函数的深度强化学习架构。该方法在不依赖外部状态信息的前提下，有效缓解环境漂移影响。我们在非平稳环境中对Deep Q-network（DQN）和Soft Actor-Critic（SAC）进行评估，展示了其在提升长期回报方面的积极作用，同时讨论了其性能局限性。
原文摘要 · Abstract (English)
Studies on rodents such as mice have shown the capabilities to adapt their behavior when dealing with changing parameters (``drift'') of the environment even if no information about change is provided (uncertainty) -- a behavior that can be modeled by forgetting mechanisms. Non-stationary Reinforcement Learning (NSRL) deals with adapting state-of-the-art RL methods to deal with changing environments: these however usually require (partially) perfect information about the drift such as ``task IDs'' or ``context''. To mitigate the effects of drift, this work develops \emph{Space-sampled Value Decay} as an explicit forgetting mechanism for value-based deep RL architectures as a simple yet effective approach. In particular we demonstrate and discuss positive effects but also limitations in achieved returns for modifications of Deep Q-networks (DQN) and Soft Actor-Critic (SAC) when evaluated on non-stationary environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。