让视觉语言动作模型在测试时动态学习环境,提升新场景适应能力。
WorldAgen: Unified State-Action Prediction with Test-Time World Model Training

- 共享Transformer架构同时预测世界状态和动作,用注意力掩码分离任务。
- 测试时通过少量探索采样和轻量更新,显著提升动作预测准确率。
- 适合需要快速适应新环境的机器人任务,尤其在部署后无数据场景下表现优。
视觉-语言-动作(VLA)模型如何适应世界动态变化的新环境?现有方法多依赖静态数据预训练,缺乏部署时主动适应机制,导致在未见场景中表现不佳。本文提出WorldAgen,一个统一框架,联合学习世界建模与动作预测,并支持测试时训练(TTT)以适应新环境。该框架采用共享Transformer主干网络,包含两个分支:(1) 世界模型头,基于历史状态-动作轨迹预测未来状态;(2) 代理模型头,根据任务指令预测动作。设计混合单向注意力掩码以分离两任务。测试时,模型生成探索性动作,收集真实状态转移数据,并进行轻量级TTT更新,从而优化世界模型,提升动作预测精度。在CALVIN与LIBERO基准上的实验表明,基线模型性能可媲美甚至超越当前最先进方法;而仅用少量样本进行TTT后,其性能更显著超越现有模型,验证了推理时动态调整世界模型的有效性。
原文摘要 · Abstract (English)
How can vision-language-action (VLA) models adapt to new environments where world dynamics shift? While recent research has combined world modeling and action prediction to improve VLA performance, existing methods largely rely on pretraining on static datasets, without mechanisms for active adaptation at deployment time. As a result, these models often fail to generalize when deployed in unseen scenarios with novel object configurations or dynamics. We present WorldAgen, a unified framework that jointly learns world modeling and action prediction while enabling Test-Time Training (TTT) to adapt to new environments. WorldAgen employs a shared Transformer backbone with two heads: (1) a world model head that predicts future states from past state-action trajectories, and (2) an agent model head that predicts actions conditioned on task instructions. We design a Mixed Unidirectional Attention Mask to separate these two models. During test time, WorldAgen samples exploratory actions, collects ground-truth state transitions, and performs lightweight TTT updates to refine its world model. This adaptation improves the model's understanding of the environment and leads to more accurate action predictions. Experiments on the CALVIN and LIBERO benchmarks demonstrate that our baseline model achieves comparable, and in some cases superior, performance to current state-of-the-art approaches. Moreover, with TTT on a small number of samples, our method surpasses existing state-of-the-art models, highlighting the effectiveness of adapting world models at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。