提出持续学习的扩散模型CoD，解决机器人任务变化时的遗忘问题。
Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
- 用经验回放机制保存旧任务数据，实现知识持续保留。
- 在90个跨领域任务上表现优于现有扩散模型和基线方法。
- 适合需要长期适应新任务的机器人控制等实际场景。
基于扩散模型的人工神经网络在游戏、控制和问答系统中表现优异，但这些系统通常面对静态数据集。而在真实场景如强化学习机器人控制中，任务是动态变化的，需按顺序连续学习。这带来了可塑性与稳定性之间的权衡：既要快速适应新任务，又要保留已有知识。为此，我们提出一种基于经验回放的持续扩散模型——持续扩散器（Continual Diffuser, CoD），赋予模型快速适应（可塑性）和长期记忆（稳定性）能力。首先构建了一个包含90个跨领域任务的离线基准测试集；随后，使用序列建模与条件生成进行决策训练；再通过小规模旧数据缓冲区回放来维持已学知识。大量实验表明，CoD在多数任务上实现了优越的可塑性-稳定性平衡，显著优于现有扩散模型及其他代表性基线方法。
原文摘要 · Abstract (English)
Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。