世界模型记得住，执行器会遗忘，用想象重练可实现持续学习。
The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL
- 冻结世界模型，用其生成的梦境进行自模仿训练，恢复遗忘技能。
- 3/3实验中，无环境交互下成功恢复技能，而传统强化学习无法恢复。
- 无需任务标签、不更新参数，可连续学习8个任务且性能稳定提升。
DreamerV3类基于模型的强化学习代理在任务序列上训练时会发生灾难性遗忘，即使使用无限容量回放缓冲区保存所有过往经验。我们提出一个持续强化学习领域长期假设但未验证的问题：哪个组件会遗忘？在未明确回放条件下，通过组件级探测（共3个种子）发现，世界模型几乎完全保留了旧任务的所有可测量信息——奖励区分能力（保持率~1.0）、价值估计和终止结构，而执行器的行为则彻底崩溃。遗忘是通道问题，而非记忆问题。通过干预验证：在世界模型冻结且想象轨迹相同的情况下，想象中的强化学习无法恢复丢失技能（0/3种子），而基于世界模型自身分级梦境的监督自模仿训练可在3/3种子中成功恢复技能，且无需环境交互。将此分级梦境重练过程穿插于训练中，构建出一种无需任务标签、参数恒定的持续学习框架：3/3四任务链均保留，普通回放仅0/3成功；3/3八任务链表现优异；优于匹配的真实回放克隆（配对差异+0.13，置换95%置信区间[0.07, 0.24]，种子完全分离）。梦境评分环节至关重要：我们识别出两种评分失败模式，提供离线检测手段提前规避，并给出实际优先的评分规则以消除缺陷。所有实验均预注册，所有被推翻的假设均如实报告。
原文摘要 · Abstract (English)
Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。