提出新方法缓解强化学习中多轮任务的遗忘问题
Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning
- 用价值函数正则化实现持续学习中的数据回放
- 在多轮任务环境中显著降低遗忘率，提升学习效率
- 适合关注长期稳定学习的强化学习研究者
数据回放已成为缓解持续强化学习（CRL）中灾难性遗忘的主要方法。然而，现有方法局限于策略梯度框架，仅对智能体（actor）进行正则化，因评论家（critic）正则化导致性能下降而被忽略。这种以智能体为中心的方法忽视了数据回放对值函数近似（value function approximation）的潜力。此外，现有评估很少考虑任务序列重复的多轮环境，这是现实场景中加剧遗忘与可塑性的关键条件。本文研究基于深度Q网络（DQN）的值函数正则化，在多轮设置下提出Qreg+NWLU：(1)连续数据回放，动态收集并更新存储的Q值；(2)“无等待”正则化，立即应用而非等待首个任务结束。两项改进在值函数近似设置下，显著提升了学习效率、遗忘缓解和知识迁移能力，优于Qreg与传统CRL方法。
原文摘要 · Abstract (English)
Data rehearsal has emerged as a leading approach for mitigating catastrophic forgetting in Continual Reinforcement Learning (CRL). However, existing work remains confined to policy gradient frameworks, regularizing only actors due to the performance degradation incurred by critic regularization. This actor-centric approach overlooks the potential of data rehearsal for value function approximation. Moreover, existing evaluations in CRL rarely consider multi-cyclic environments where task sequences repeat, a critical real-world scenario that exacerbates forgetting and plasticity. We investigate data rehearsal for Deep Q-Networks using Q-value regularization in multi-cyclic settings and propose Qreg+NWLU which introduces two simple modifications: (1) continuous data rehearsal that dynamically collects and updates stored Q-values throughout training, and (2) "No-Wait" regularization that applies immediately rather than after the first task. Together, these modifications yield improvements in learning efficiency, forgetting mitigation, and knowledge transfer over Qreg and conventional CRL methods within value function approximation settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。