让智能体无任务标签地持续学习，靠内在动机自动积累经验。
Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation
- 通过无任务探索积累内在动机知识，实现自监督学习。
- 在多个任务上比基线方法少用30%~50%样本完成学习。
- 适合无标签、无明确任务边界的持续学习场景。
构建通用学习系统的关键在于，当新数据到来时无需从头训练即可解决多个任务。这至关重要，因为每个任务都需要大量训练时间。持续学习面临多重挑战：（1）防止灾难性遗忘以保留旧知识；（2）实现正向迁移以加速新任务学习；（3）支持大规模任务扩展；（4）在无任务标签且任务边界模糊的情况下仍能有效学习。本文提出任务无关策略蒸馏（TAPD）框架，通过引入任务无关探索阶段，智能体在无外部目标下仅最大化内在动机进行环境探索，并将所获知识用于后续蒸馏。该过程使智能体以自监督方式系统性发现新状态。利用任务无关蒸馏知识后，智能体能更高效地完成下游任务，显著提升样本效率。代码已开源：https://github.com/wabbajack1/TAPD。
原文摘要 · Abstract (English)
Central to the development of universal learning systems is the ability to solve multiple tasks without retraining from scratch when new data arrives. This is crucial because each task requires significant training time. Addressing the problem of continual learning necessitates various methods due to the complexity of the problem space. This problem space includes: (1) addressing catastrophic forgetting to retain previously learned tasks, (2) demonstrating positive forward transfer for faster learning, (3) ensuring scalability across numerous tasks, and (4) facilitating learning without requiring task labels, even in the absence of clear task boundaries. In this paper, the Task-Agnostic Policy Distillation (TAPD) framework is introduced. This framework alleviates problems (1)-(4) by incorporating a task-agnostic phase, where an agent explores its environment without any external goal and maximizes only its intrinsic motivation. The knowledge gained during this phase is later distilled for further exploration. Therefore, the agent acts in a self-supervised manner by systematically seeking novel states. By utilizing task-agnostic distilled knowledge, the agent can solve downstream tasks more efficiently, leading to improved sample efficiency. Our code is available at the repository: https://github.com/wabbajack1/TAPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。