提出SPHERE方法，解决MoE在持续强化学习中能力退化问题。
SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning
- 基于NTK理论定义谱可塑性退化，设计可计算的代理指标
- 在MetaWorld和HumanoidBench上提升成功率133%和50%
- 适合持续学习、需长期适应新任务的强化学习系统
在深度强化学习中，智能体从经验流中训练。持续学习场景下，其学习新技能的能力会随训练过程下降，称为可塑性损失。尽管混合专家（MoE）网络能支持规模定律并学习多样化技能，但在持续强化学习中性能仍会退化，表明可塑性丧失。本文基于神经正切核（NTK）理论，将MoE策略中的可塑性损失形式化为谱可塑性损失，并推导出可计算的谱可塑性代理指标，该指标仅依赖于各专家特征矩阵。基于此，我们提出针对MoE策略的实用解析范数（Parseval penalty）——SPHERE，有效缓解谱可塑性损失。在MetaWorld和HumanoidBench上，SPHERE相较于无正则化的MoE基线，平均成功率达133%和50%的提升，且全程保持更高谱可塑性。
原文摘要 · Abstract (English)
In deep reinforcement learning (DRL), an agent is trained from a stream of experience. In a continual learning setting, such agents can suffer from plasticity loss: their ability to learn new skills from new experiences diminishes over training. Recently, Mixture-of-Experts (MoE) networks have been reported to enable scaling laws and facilitate the learning of diverse skills. However, in continual reinforcement learning settings, their performance can degenerate as learning proceeds, indicating a loss of plasticity. To address this, building on Neural Tangent Kernel (NTK) theory, we formalize the plasticity loss in MoE policies as a loss of spectral plasticity. We then derive a tractable proxy for spectral plasticity, one expressible in terms of individual expert feature matrices. Leveraging this proxy, we introduce SPHERE, a practical Parseval penalty tailored for MoE-based policies that alleviates the loss of spectral plasticity. On MetaWorld and HumanoidBench, SPHERE improves average success under continual RL by 133% and 50% over an unregularized MoE baseline, while maintaining higher spectral plasticity throughout training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。