提出遗忘与增长策略，提升强化学习在连续控制中的样本效率。
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
- 通过渐进衰减早期经验，缓解记忆过拟合问题。
- 训练中动态扩展网络参数，增强对数据模式的捕捉能力。
- 在40多个任务上优于当前最优算法，适合复杂控制场景。
连续控制的深度强化学习近期取得显著进展。然而，现有方法常受首因偏差影响，过度拟合存储在回放缓冲区中的早期经验，限制了智能体的样本效率和泛化能力。人类则较少受此类偏差影响，部分原因在于婴儿期遗忘现象：新神经元的形成会破坏早期记忆痕迹，导致初始经验被遗忘。受神经科学中遗忘与生长双重机制的启发，本文提出一种名为「遗忘与增长」（FoG）的新深度强化学习算法，包含两个机制：一是经验回放衰减（ER Decay），通过逐渐降低早期经验的影响来平衡记忆；二是网络扩展，通过在训练过程中动态添加新参数，增强智能体利用已有数据模式的能力。在四个主流连续控制基准测试、超过40个任务上的实验证明，FoG在性能上优于当前最先进算法，包括BRO、SimBa和TD-MPC2。
原文摘要 · Abstract (English)
Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias, a tendency to overfit early experiences stored in the replay buffer, which limits an RL agent's sample efficiency and generalizability. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose Forget and Grow (FoG), a new deep RL algorithm with two mechanisms introduced. First, Experience Replay Decay (ER Decay) "forgetting early experience", which balances memory by gradually reducing the influence of early experiences. Second, Network Expansion, "growing neural capacity", which enhances agents' capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of FoG against SoTA existing deep RL algorithms, including BRO, SimBa, and TD-MPC2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。