用历史任务的值函数加速深度强化学习,提升初期学习效率。
Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning
- 从过往任务提取紧凑表格值函数作为知识库
- 在未探索区域软融合历史值,逐步过渡到当前学习结果
- 无需策略或示范,适合连续控制任务的快速启动
值函数初始化(VFI)通过利用先前任务的值估计,在强化学习中实现跳起式启动。尽管该方法在表格设置中已成熟,但在深度强化学习(DRL)中面临状态-动作空间连续、神经网络近似噪声大、难以存储所有历史模型等问题。本文提出DQInit,一种适用于DRL的值函数初始化方法。DQInit复用先前解决任务中提取的紧凑表格Q值作为可迁移知识库,采用基于已知度的机制,在未探索区域软性融入这些转移值,并渐进转向代理自身学习的估计,避免固定时间衰减的局限。该方法仅依赖值估计而非策略或演示,提供了一种新视角的知识迁移方式,有效结合了跳起式RL与策略蒸馏的优势,同时规避其缺陷。在多个连续控制任务上的实验表明,相较于标准初始化和现有迁移技术,DQInit在早期学习效率、稳定性和整体性能上均显著提升。
原文摘要 · Abstract (English)
Value function initialization (VFI) is an effective way to achieve a jumpstart in reinforcement learning (RL) by leveraging value estimates from prior tasks. While this approach is well established in tabular settings, extending it to deep reinforcement learning (DRL) poses challenges due to the continuous nature of the state-action space, the noisy approximations of neural networks, and the impracticality of storing all past models for reuse. In this work, we address these challenges and introduce DQInit, a method that adapts value function initialization to DRL. DQInit reuses compact tabular Q-values extracted from previously solved tasks as a transferable knowledge base. It employs a knownness-based mechanism to softly integrate these transferred values into underexplored regions and gradually shift toward the agent's learned estimates, avoiding the limitations of fixed time decay. Our approach offers a novel perspective on knowledge transfer in DRL by relying solely on value estimates rather than policies or demonstrations, effectively combining the strengths of jumpstart RL and policy distillation while mitigating their drawbacks. Experiments across multiple continuous control tasks demonstrate that DQInit consistently improves early learning efficiency, stability, and overall performance compared to standard initialization and existing transfer techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。