arXiv:2604.15414cs.LGcs.AI2026-04

提出多策略存档机制,让持续强化学习更灵活适应新任务。

Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

论文配图:Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
图 1 · 摘自论文原文
  • 构建任务专属的多样化策略存档,共享潜在空间保持可比性。
  • 在干扰后恢复旧任务速度更快,连续学习成功率提升显著。
  • 适合需要长期适应能力的智能体开发,如机器人、自动驾驶。

持续强化学习需平衡记忆与适应,但多数方法依赖单一策略保存,导致策略失效后难以快速重学,出现‘可塑性丢失’问题。受质量-多样性方法启发,本文提出TeLAPA(Transfer-Enabled Latent-Aligned Policy Archives)框架,将行为多样化的策略聚类为每任务存档,并维护共享潜在空间,使存档策略在非平稳变化下仍可比较与复用。该框架从保留孤立解转向维持技能对齐的策略邻域,支持未来高效再学习。在MiniGrid持续学习设置中,TeLAPA成功学习更多任务,干扰后恢复旧任务更快,全程性能更高。分析表明,局部最优策略未必是迁移最优,有效复用依赖于保留并选择多个近邻替代方案,而非压缩为单一代表。研究重新定义了持续学习的核心目标:可复用且高效的策略邻域。

原文摘要 · Abstract (English)

Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on \emph{single-model preservation}, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of \emph{loss of plasticity} that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce \textsc{TeLAPA} (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining \emph{skill-aligned neighborhoods} with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, \textsc{TeLAPA} learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.

持续学习强化学习策略存档可塑性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。