提出新方法让强化学习更高效且适应不完全观测环境。
Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning
- 扩展DAE理论至部分可观测场景，仅微小改动
- 用离散隐状态模型近似转移概率，降低计算开销
- 在雅达利游戏上验证，样本效率高且可扩展
直接优势估计（DAE）已被证明能提升深度强化学习的样本效率。然而，其依赖完整环境观测限制了在真实场景的应用，且建模转移概率带来高维观测下的巨大计算开销。本文针对这两点改进：首先，将DAE理论框架扩展至部分可观测域，仅需最小修改；其次，引入离散隐动态模型，高效近似转移概率以降低计算复杂度。在雅达利学习环境上的实验表明，该方法能随函数逼近器容量有效扩展，同时保持高样本效率。
原文摘要 · Abstract (English)
Direct Advantage Estimation (DAE) has been shown to improve the sample efficiency of deep reinforcement learning algorithms. However, its reliance on full environment observability limits its applicability in realistic settings, and its requirement to model transition probabilities incurs substantial computational overhead for high-dimensional observations. In the present work, we address both limitations. First, we extend the theoretical framework of DAE to partially observable domains with minimal modifications. Second, we reduce its computational complexity by introducing discrete latent dynamics models that efficiently approximate transition probabilities. We evaluate our approach on the Arcade Learning Environment and find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。