用事件驱动的变分采样提升强化学习探索效率
EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning
- 引入事件感知卷积层，结合高斯丢弃实现变分贝叶斯采样
- 在10万次Atari游戏测试中显著提升策略学习效果
- 适合物体基础环境下的模型化强化学习任务
基于后验采样的强化学习（PSRL）是模型基础强化学习（MBRL）的重要方法，通过汤普森采样对环境转移动态和奖励函数进行后验建模。然而，在高维状态与动作空间下，后验分布难以计算。近期研究发现，神经网络中的丢弃机制可诱导出近似后验的变分分布。本文提出事件驱动的变分探索方法（EVaDE），专为物体基础领域设计，利用对象域先验知识构建三类事件感知卷积层，以引导探索。这些层采用高斯丢弃并嵌入深度神经网络中间，支持变分汤普森采样。实验表明，搭载EVaDE的模拟策略学习算法（EVaDE-SimPLe）在10万次Atari游戏套件上表现优异。
原文摘要 · Abstract (English)
Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function, which are intractable for tasks with high-dimensional state and action spaces. Recent works show that dropout, used in conjunction with neural networks, induces variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), which are variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration. These layers rely on Gaussian dropouts and are inserted between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE-equipped Simulated Policy Learning (EVaDE-SimPLe) on the 100K Atari game suite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。