用分布式强化学习教师持续蒸馏，实现稳定高效终身学习。
Continual Policy Distillation from Distributed Reinforcement Learning Teachers
- 分阶段训练单任务教师模型，再统一蒸馏到通用学生模型
- 在Meta-World上恢复超85%教师性能，任务遗忘率低于10%
- 结合专家混合与回放机制，兼顾学习稳定性与适应性
持续强化学习旨在构建能跨多样化任务持续获取知识并缓解灾难性遗忘的终身学习智能体。这需要有效平衡稳定性与可塑性，并利用已有经验快速泛化至新任务。尽管已有多种增强策略被提出，但直接将强化学习应用于连续任务流仍面临可扩展性挑战。本文提出一种新型师生框架，将持续强化学习解耦为两个独立过程：通过分布式强化学习训练单任务教师模型，并持续将其蒸馏至一个中心通用模型。该设计基于观察：强化学习擅长解决单个任务，而策略蒸馏——一种相对稳定的监督学习过程——与大型基础模型和多任务学习高度契合。此外，采用专家混合（MoE）架构和基于回放的方法以提升持续策略蒸馏过程的可塑性与稳定性。在Meta-World基准上的大量实验表明，该框架实现了高效的持续强化学习，在恢复超过85%教师性能的同时，将任务级遗忘控制在10%以内。
原文摘要 · Abstract (English)
Continual Reinforcement Learning (CRL) aims to develop lifelong learning agents to continuously acquire knowledge across diverse tasks while mitigating catastrophic forgetting. This requires efficiently managing the stability-plasticity dilemma and leveraging prior experience to rapidly generalize to novel tasks. While various enhancement strategies for both aspects have been proposed, achieving scalable performance by directly applying RL to sequential task streams remains challenging. In this paper, we propose a novel teacher-student framework that decouples CRL into two independent processes: training single-task teacher models through distributed RL and continually distilling them into a central generalist model. This design is motivated by the observation that RL excels at solving single tasks, while policy distillation -- a relatively stable supervised learning process -- is well aligned with large foundation models and multi-task learning. Moreover, a mixture-of-experts (MoE) architecture and a replay-based approach are employed to enhance the plasticity and stability of the continual policy distillation process. Extensive experiments on the Meta-World benchmark demonstrate that our framework enables efficient continual RL, recovering over 85% of teacher performance while constraining task-wise forgetting to within 10%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。