用冻结的几何特征预测未来,让世界模型更准更快
VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
- 用VGGT的潜空间特征做世界状态,通过轻量Transformer预测其演变轨迹
- 在KITTI等数据集上深度预测精度显著领先,速度比基线快3.6-5倍
- 适合需要高效3D场景预测的自动驾驶与机器人研究者
现有世界模型主要聚焦于生成未来视频帧的像素细节,但预测结果常出现几何不一致问题。本文提出VGGT-World,一种不生成视频的几何世界模型,直接预测冻结的几何基础模型(GFM)特征随时间的演变。具体地,将冻结VGGT的潜变量作为世界状态,并训练一个轻量级时序流Transformer进行自回归预测。在高维(d=1024)特征空间中面临两大挑战:(i) 标准速度预测流匹配失效,(ii) 自回归滚动受累积暴露偏差影响。我们通过清洁目标(z-预测)参数化提升信噪比,采用两阶段潜流强制课程训练,逐步让模型依赖自身部分去噪的滚动输出。在KITTI、Cityscapes和TartanAir上的实验表明,VGGT-World在深度预测上显著优于最强基线,运行速度提升3.6–5倍,仅需0.43B可训练参数,证明冻结的GFM特征是高效且有效的3D世界建模状态。
原文摘要 · Abstract (English)
World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。