通过交互数据估算因果差异,让强化学习课程更高效。
Causal-Paced Deep Reinforcement Learning
- 基于交互数据近似任务间因果结构差异,动态生成课程
- 在点质量、双足行走等任务上收敛更快、回报更高
- 适合追求样本效率和结构感知的强化学习研究者
设计有效的任务序列对课程强化学习(CRL)至关重要,要求智能体通过中间任务逐步习得技能。当前关键挑战在于识别能促进探索且具备良好迁移性的任务。已有方法依赖真实因果结构(SCM),但现实中难以获取。本文提出因果驱动的深度强化学习(CP-DRL),利用交互数据近似任务间的因果模型差异,该信号捕捉任务新颖性,并与智能体的学习能力(以奖励增长衡量)结合,形成统一目标。实验表明,CP-DRL在点质量基准上收敛更快、回报更高;在双足行走-简单设置中降低方差,且在不可行变体中达到最高平均性能。结果表明,利用任务间的因果关系可提升课程学习的结构感知力与样本效率。完整实现代码见 https://github.com/Cho-Geonwoo/CP-DRL。
原文摘要 · Abstract (English)
Designing effective task sequences is crucial for curriculum reinforcement learning (CRL), where agents must gradually acquire skills by training on intermediate tasks. A key challenge in CRL is to identify tasks that promote exploration, yet are similar enough to support effective transfer. While recent approach suggests comparing tasks via their Structural Causal Models (SCMs), the method requires access to ground-truth causal structures, an unrealistic assumption in most RL settings. In this work, we propose Causal-Paced Deep Reinforcement Learning (CP-DRL), a curriculum learning framework aware of SCM differences between tasks based on interaction data approximation. This signal captures task novelty, which we combine with the agent's learnability, measured by reward gain, to form a unified objective. Empirically, CP-DRL outperforms existing curriculum methods on the Point Mass benchmark, achieving faster convergence and higher returns. CP-DRL demonstrates reduced variance with comparable final returns in the Bipedal Walker-Trivial setting, and achieves the highest average performance in the Infeasible variant. These results indicate that leveraging causal relationships between tasks can improve the structure-awareness and sample efficiency of curriculum reinforcement learning. We provide the full implementation of CP-DRL to facilitate the reproduction of our main results at https://github.com/Cho-Geonwoo/CP-DRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。