解决多智能体强化学习中神经元僵化导致适应力下降的问题
Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods
- 识别出梯度更新极小的'僵化神经元'是学习停滞主因
- 新方法KNIFE用三组件替换僵化神经元，提升适应能力
- 在SMACv2等任务上显著优于现有方法，适合复杂协作场景
多智能体强化学习的价值分解方法在迁移至新任务时常出现适应力下降问题。我们发现其根源在于'僵化神经元'——这类神经元的梯度更新相对于权重几乎为零，阻碍了学习。尽管已有注入塑性方法，但对僵化神经元无效。为此，我们提出知识保持型神经元级塑性聚焦注入方法（KNIFE），直接针对僵化神经元：用由三个组件构成的复合单元替代原神经元——冻结的知识神经元保留已学知识，重初始化的活跃神经元恢复学习能力，补偿神经元确保输出与原结果一致，从而维持原有合作策略。在SMACv2、捕食者-猎物和矩阵博弈上的大量实验表明，KNIFE显著优于当前最优塑性注入方法。
原文摘要 · Abstract (English)
Multi-Agent Reinforcement Learning (MARL) value factorization methods can suffer from a loss of plasticity, gradually failing to adapt when transferring to new task instances. We trace this issue to stagnant neurons, units whose gradient updates become negligibly small relative to their weights, thereby hindering learning. While existing plasticity injection methods exist, they prove ineffective for such neurons. To address this, we propose Knowledge-retentive Neuron-level PlastIcity Focusing InjEction (KNIFE), a novel method that directly targets stagnant neurons. KNIFE replaces each stagnant neuron with a composite unit comprising three specialized components: a frozen knowledge neuron to preserve acquired knowledge, a re-initialized active neuron to restore learning capacity, and a compensation neuron to ensure the combined output matches the original, thus maintaining previous learned cooperation knowledge. Extensive experiments on SMACv2, predator-prey, and matrix games demonstrate that KNIFE significantly outperforms state-of-the-art plasticity injection methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。