用多样化DQN组合与仲裁机制缓解强化学习持续遗忘问题。
An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learning
- 构建多样化的DQN模型集合，通过差异化的价值函数增强泛化能力。
- 基于近期表现可靠性动态选择最优代理，显著提升连续学习性能。
- 适合研究持续强化学习、脑启发智能系统的科研人员参考。
深度强化学习（RL）在静态环境中能高效学习最优策略，但在持续强化学习（CRL）场景中易出现灾难性遗忘，导致先前知识丢失。为解决该问题，本文提出一种基于多样化DQN变体的集成仲裁控制机制（ACED-DQN）。受前额叶皮层多代理协同决策启发，模型融合两大核心思想：（1）训练一组具有差异化价值函数的DQN变体；（2）引入仲裁控制，优先选择近期误差更小、可靠性更高的代理。在静态与持续环境中的实验证明，该框架显著提升了学习稳定性与性能。本工作构建了一种受人类大脑启发的持续学习框架，使RL代理具备持续适应新任务的能力。
原文摘要 · Abstract (English)
Deep reinforcement learning (RL) models, despite their efficiency in learning an optimal policy in static environments, easily loses previously learned knowledge (i.e., catastrophic forgetting). It leads RL models to poor performance in continual reinforcement learning (CRL) scenarios. To address this, we present an arbitration control mechanism over an ensemble of RL agents. It is motivated by and closely aligned with how humans make decisions in a CRL context using an arbitration control of multiple RL agents in parallel as observed in the prefrontal cortex. We integrated two key ideas into our model: (1) an ensemble of RLs (i.e., DQN variants) explicitly trained to have diverse value functions and (2) an arbitration control that prioritizes agents with higher reliability (i.e., less error) in recent trials. We propose a framework for CRL, an Arbitration Control for an Ensemble of Diversified DQN variants (ACED-DQN). We demonstrate significant performance improvements in both static and continual environments, supported by empirical evidence showing the effectiveness of arbitration control over diversified DQNs during training. In this work, we introduced a framework that enables RL agents to continuously learn, with inspiration from the human brain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。