arXiv:2505.10819cs.AIcs.LG2025-05NeurIPS被引 24

用程序专家组合建模复杂世界,少样本也能高效学习。

PoE-World: Compositional World Modeling with Products of Programmatic Experts

  • 将世界模型构造成由大模型生成的程序专家加权产物
  • 仅需少量观测即可学习复杂随机世界模型
  • 适合需要少样本泛化能力的强化学习与智能体研究

理解世界运作机制是构建能适应复杂环境的AI智能体的核心。传统基于深度学习的世界模型依赖大量训练数据,且难以从稀疏观测中灵活更新知识。近期利用大语言模型(LLMs)进行程序合成的方法提供了一种新思路:以源代码形式表示世界模型,支持从极少数据中实现强泛化。然而,现有程序结构世界模型的应用仍局限于自然语言和网格世界。本文提出一种新型程序合成方法——PoE-World,通过指数加权的程序专家组合来有效建模复杂非网格世界。我们证明该方法仅需少数观测即可学习复杂的随机世界模型。通过将所学模型嵌入基于模型的规划智能体,在Atari的Pong和Montezuma's Revenge上实现了高效性能与未见关卡的泛化能力。代码及模型演示视频已公开于https://topwasu.github.io/poe-world。

原文摘要 · Abstract (English)

Learning how the world works is central to building AI agents that can adapt to complex environments. Traditional world models based on deep learning demand vast amounts of training data, and do not flexibly update their knowledge from sparse observations. Recent advances in program synthesis using Large Language Models (LLMs) give an alternate approach which learns world models represented as source code, supporting strong generalization from little data. To date, application of program-structured world models remains limited to natural language and grid-world domains. We introduce a novel program synthesis method for effectively modeling complex, non-gridworld domains by representing a world model as an exponentially-weighted product of programmatic experts (PoE-World) synthesized by LLMs. We show that this approach can learn complex, stochastic world models from just a few observations. We evaluate the learned world models by embedding them in a model-based planning agent, demonstrating efficient performance and generalization to unseen levels on Atari's Pong and Montezuma's Revenge. We release our code and display the learned world models and videos of the agent's gameplay at https://topwasu.github.io/poe-world.

世界模型程序合成少样本学习强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。