arXiv:2409.14084cs.LGcs.AI2024-09

仅用合成数据训练的单次适应世界模型,可快速适配新环境。

One-shot World Models Using a Transformer Trained on a Synthetic Prior

  • 基于合成先验数据,用Transformer实现上下文学习的世界模型。
  • 仅需1000步轨迹即可在网格世界、CartPole等环境中快速适应。
  • 适合研究纯合成数据训练世界模型的学者,推动通用智能体发展。

世界模型是现实环境的空间时间压缩表示,可用于训练智能体或执行规划。然而,传统世界模型通常在真实环境观测上训练,难以迁移到其他真实环境。我们提出单次世界模型(OSWM),一种通过纯合成数据以上下文学习方式训练的Transformer世界模型。其先验由多个随机初始化的神经网络构成,每个网络模拟目标环境的状态与奖励维度动态。采用监督学习方法,随机遮蔽部分状态和奖励,让OSWM基于剩余上下文进行概率预测。推理时,仅需提供1000步过渡数据作为上下文,即可快速适应简单网格世界、CartPole Gym及自定义控制环境,并成功训练出解决环境的策略。但向更复杂环境迁移仍具挑战。尽管如此,该工作为完全从合成数据学习世界模型提供了重要进展。

原文摘要 · Abstract (English)

A World Model is a compressed spatial and temporal representation of a real world environment that allows one to train an agent or execute planning methods. However, world models are typically trained on observations from the real world environment, and they usually do not enable learning policies for other real environments. We propose One-Shot World Model (OSWM), a transformer world model that is learned in an in-context learning fashion from purely synthetic data sampled from a prior distribution. Our prior is composed of multiple randomly initialized neural networks, where each network models the dynamics of each state and reward dimension of a desired target environment. We adopt the supervised learning procedure of Prior-Fitted Networks by masking next-state and reward at random context positions and query OSWM to make probabilistic predictions based on the remaining transition context. During inference time, OSWM is able to quickly adapt to the dynamics of a simple grid world, as well as the CartPole gym and a custom control environment by providing 1k transition steps as context and is then able to successfully train environment-solving agent policies. However, transferring to more complex environments remains a challenge, currently. Despite these limitations, we see this work as an important stepping-stone in the pursuit of learning world models purely from synthetic data.

世界模型合成数据上下文学习单次适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。