用掩码生成先验提升世界模型序列建模能力,显著改善强化学习表现。
Masked Generative Priors Improve World Models Sequence Modelling Capabilities
- 用MaskGIT替代传统MLP先验,构建更优的序列生成机制。
- 在Atari 100k上强化学习性能大幅提升,首次成功应用于连续动作环境。
- 适合关注世界模型、生成式强化学习的研究者与开发者。
深度强化学习已成为复杂环境中构建智能体的主流方法。基于模型的方法通过世界模型预测环境动态,显著提升数据效率,是连接研究与真实应用的关键。世界模型通过自监督方式训练生成序列模型,在想象中学习以提高样本效率。近期,掩码生成建模展现出更高效且优越的归纳偏置,用于序列建模与生成。我们基于高效的随机变压器世界模型(STORM)架构,将传统MLP先验替换为掩码生成先验(如MaskGIT),提出GIT-STORM。在两个下游任务上评估:强化学习和视频预测。GIT-STORM在Atari 100k基准上的强化学习任务中表现显著提升。此外,首次将基于Transformer的世界模型应用于连续动作环境,填补了先前研究的空白。为此,我们引入状态混合函数,融合隐状态表示与动作,使模型可处理连续控制任务。在DeepMind Control Suite上通过定性与定量分析验证该方法的有效性,证明了基于Transformer的世界模型在此新领域的潜力。结果表明,MaskGIT动力学先验具有高度通用性和有效性,为更精准的世界模型与高效强化学习策略铺平道路。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (RL) has become the leading approach for creating artificial agents in complex environments. Model-based approaches, which are RL methods with world models that predict environment dynamics, are among the most promising directions for improving data efficiency, forming a critical step toward bridging the gap between research and real-world deployment. In particular, world models enhance sample efficiency by learning in imagination, which involves training a generative sequence model of the environment in a self-supervised manner. Recently, Masked Generative Modelling has emerged as a more efficient and superior inductive bias for modelling and generating token sequences. Building on the Efficient Stochastic Transformer-based World Models (STORM) architecture, we replace the traditional MLP prior with a Masked Generative Prior (e.g., MaskGIT Prior) and introduce GIT-STORM. We evaluate our model on two downstream tasks: reinforcement learning and video prediction. GIT-STORM demonstrates substantial performance gains in RL tasks on the Atari 100k benchmark. Moreover, we apply Transformer-based World Models to continuous action environments for the first time, addressing a significant gap in prior research. To achieve this, we employ a state mixer function that integrates latent state representations with actions, enabling our model to handle continuous control tasks. We validate this approach through qualitative and quantitative analyses on the DeepMind Control Suite, showcasing the effectiveness of Transformer-based World Models in this new domain. Our results highlight the versatility and efficacy of the MaskGIT dynamics prior, paving the way for more accurate world models and effective RL policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。