提出新方法提升动态环境下的强化学习采样效率。
Sample Efficient Experience Replay in Non-stationary Environments
- 用环境动态差异度量分离环境变化与策略更新影响
- 在4个动态基准上使算法性能提升11.54%
- 适合研究动态环境强化学习的学者和工程师
非平稳环境中的强化学习面临挑战，因动态变化和奖励调整会迅速使历史经验失效。传统经验回放方法（尤其是基于TD误差优先级的）难以区分环境变化与策略更新带来的影响，导致动态条件下学习效率低下。为此，我们提出环境动态差异度量（DoE），用于隔离环境漂移对价值函数的影响。基于此，我们设计了环境差异优先经验回放（DEER），一种自适应经验回放框架，根据策略更新和环境变化双重信号进行过渡优先级排序。DEER采用二分类器检测环境突变，并在每次突变前后应用不同优先级策略，实现更高效的样本利用。在四个非平稳基准测试中，相较于当前最优的经验回放方法，DEER使离策略算法性能平均提升11.54%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) in non-stationary environments is challenging, as changing dynamics and rewards quickly make past experiences outdated. Traditional experience replay (ER) methods, especially those using TD-error prioritization, struggle to distinguish between changes caused by the agent's policy and those from the environment, resulting in inefficient learning under dynamic conditions. To address this challenge, we propose the Discrepancy of Environment Dynamics (DoE), a metric that isolates the effects of environment shifts on value functions. Building on this, we introduce Discrepancy of Environment Prioritized Experience Replay (DEER), an adaptive ER framework that prioritizes transitions based on both policy updates and environmental changes. DEER uses a binary classifier to detect environment changes and applies distinct prioritization strategies before and after each shift, enabling more sample-efficient learning. Experiments on four non-stationary benchmarks demonstrate that DEER further improves the performance of off-policy algorithms by 11.54 percent compared to the best-performing state-of-the-art ER methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。