用概率框架提升持续学习中智能体的泛化能力与适应速度
Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theory
- 基于PAC-Bayes理论构建共享策略分布，实现快速新任务适应
- 理论证明保留越多历史任务，泛化性能越好，样本复杂度可量化
- 适合追求理论保障的持续强化学习研究者与系统设计者
持续强化学习（Lifelong RL）旨在将单任务RL扩展至更真实的动态环境。本文提出EPIC（Empirical PAC-Bayes that Improves Continuously）算法，基于PAC-Bayes理论构建共享策略分布——世界策略（world policy），使智能体在新任务中快速适应并保留过往经验。理论分析建立了算法泛化性能与记忆中保留先验任务数量之间的关系，并推导出以RL遗憾（regret）形式表示的样本复杂度。在多种环境上的大量实验表明，EPIC显著优于现有方法，在提供理论保证的同时兼具实际有效性。
原文摘要 · Abstract (English)
Lifelong reinforcement learning (RL) has been developed as a paradigm for extending single-task RL to more realistic, dynamic settings. In lifelong RL, the "life" of an RL agent is modeled as a stream of tasks drawn from a task distribution. We propose EPIC (Empirical PAC-Bayes that Improves Continuously), a novel algorithm designed for lifelong RL using PAC-Bayes theory. EPIC learns a shared policy distribution, referred to as the world policy, which enables rapid adaptation to new tasks while retaining valuable knowledge from previous experiences. Our theoretical analysis establishes a relationship between the algorithm's generalization performance and the number of prior tasks preserved in memory. We also derive the sample complexity of EPIC in terms of RL regret. Extensive experiments on a variety of environments demonstrate that EPIC significantly outperforms existing methods in lifelong RL, offering both theoretical guarantees and practical efficacy through the use of the world policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。