arXiv:2511.22904cs.CLcs.LG2025-11

用语言描述环境动态,让智能体在未知游戏中表现更好。

Language-conditioned world model improves policy generalization by reading environmental descriptions

  • 用注意力机制将语言描述与观测中的实体对齐,提升世界模型理解力。
  • 无需专家示范或推理时规划,在新游戏上泛化能力显著优于基线。
  • 可通过合成轨迹微调,部署前优化策略,适合实际应用落地。

为在真实世界中有效与人类交互,智能体需理解描述环境动态的语言——即环境如何运作,而不仅仅是任务指令“做什么”。理解这类动态描述对人机交互和智能体行为至关重要。现有基于模型的方法将语言融入世界模型以学习行为策略,但大多未展示对未见游戏的泛化能力,或依赖受限假设,如可接受推理延迟或需专家示范。本文提出一种无需这些假设的模型强化学习方法:通过与环境交互训练语言条件世界模型,并从中学习策略,不依赖规划或专家数据。提出基于DreamerV3的语义感知编码器(LED-WM),其观察编码器使用注意力机制显式将语言描述与观测中的实体对齐。实验表明,在MESSENGER和MESSENGER-WM两个环境中,该方法在多种设置下,对由新颖动态和语言描述的未见游戏,策略泛化能力显著优于其他基线。进一步验证了部署前可通过世界模型生成的合成测试轨迹对策略进行微调,提升性能。

原文摘要 · Abstract (English)

To interact effectively with humans in the real world, it is important for agents to understand language that describes the dynamics of the environment--that is, how the environment behaves--rather than just task instructions specifying "what to do". Understanding this dynamics-descriptive language is important for human-agent interaction and agent behavior. Recent work address this problem using a model-based approach: language is incorporated into a world model, which is then used to learn a behavior policy. However, these existing methods either do not demonstrate policy generalization to unseen games or rely on limiting assumptions. For instance, assuming that the latency induced by inference-time planning is tolerable for the target task or expert demonstrations are available. Expanding on this line of research, we focus on improving policy generalization from a language-conditioned world model while dropping these assumptions. We propose a model-based reinforcement learning approach, where a language-conditioned world model is trained through interaction with the environment, and a policy is learned from this model--without planning or expert demonstrations. Our method proposes Language-aware Encoder for Dreamer World Model (LED-WM) built on top of DreamerV3. LED-WM features an observation encoder that uses an attention mechanism to explicitly ground language descriptions to entities in the observation. We show that policies trained with LED-WM generalize more effectively to unseen games described by novel dynamics and language compared to other baselines in several settings in two environments: MESSENGER and MESSENGER-WM.To highlight how the policy can leverage the trained world model before real-world deployment, we demonstrate the policy can be improved through fine-tuning on synthetic test trajectories generated by the world model.

世界模型语言理解策略泛化强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。