让大模型从无动作标签的序列中自学习世界模型，提升推理与规划能力。
Self-Improving World Modelling with Latent Actions
- 将动作视为隐变量，交替优化正向与逆向世界模型。
- 在多个基准上提升14%至28%，最高达28%性能增益。
- 适合需要自主建模的视觉语言模型与大模型应用。
内部世界建模——即在给定先前状态X和动作Z时预测下一状态Y——对大语言模型和视觉语言模型的推理与规划至关重要。传统方法需依赖昂贵的动作标注轨迹。我们提出SWIRL，一种自改进框架，仅使用状态序列即可学习，将动作视为潜在变量，并交替进行前向世界建模（FWM）P_θ(Y|X,Z)与逆动力学建模（IDM）Q_ϕ(Z|X,Y)。SWIRL迭代两个阶段：(1) 变分信息最大化，更新FWM以使下一状态在给定先验状态条件下与潜在动作的最大化条件互信息，促进可识别一致性；(2) ELBO最大化，更新IDM以解释观测到的状态转移，实现坐标上升。两模型均采用强化学习（具体为GRPO）训练，对方模型的对数概率作为奖励信号。我们提供了两种更新的理论可学习性保证，并在多个环境评估了SWIRL在大模型与视觉语言模型上的表现：单轮与多轮开放世界视觉动态及合成文本环境，涵盖物理、网页与工具调用任务。SWIRL在AURORABench上提升16%，ByteMorph上提升28%，WorldPredictionBench上提升16%，StableToolBench上提升14%。
原文摘要 · Abstract (English)
Internal modelling of the world -- predicting transitions between previous states $X$ and next states $Y$ under actions $Z$ -- is essential to reasoning and planning for LLMs and VLMs. Learning such models typically requires costly action-labelled trajectories. We propose SWIRL, a self-improvement framework that learns from state-only sequences by treating actions as a latent variable and alternating between Forward World Modelling (FWM) $P_θ(Y|X,Z)$ and an Inverse Dynamics Modelling (IDM) $Q_ϕ(Z|X,Y)$. SWIRL iterates two phases: (1) Variational Information Maximisation, which updates the FWM to generate next states that maximise conditional mutual information with latent actions given prior states, encouraging identifiable consistency; and (2) ELBO Maximisation, which updates the IDM to explain observed transitions, effectively performing coordinate ascent. Both models are trained with reinforcement learning (specifically, GRPO) with the opposite frozen model's log-probability as a reward signal. We provide theoretical learnability guarantees for both updates, and evaluate SWIRL on LLMs and VLMs across multiple environments: single-turn and multi-turn open-world visual dynamics and synthetic textual environments for physics, web, and tool calling. SWIRL achieves gains of 16% on AURORABench, 28% on ByteMorph, 16% on WorldPredictionBench, and 14% on StableToolBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。