arXiv:2512.11225cs.CVcs.AI2025-12被引 5

用视觉基础模型特征生成未来世界,更准更快且可直接用于决策。

VFMF: World Modeling by Forecasting Vision Foundation Model Features

  • 在视觉基础模型特征空间中进行自回归流匹配,实现生成式预测。
  • 相比确定性回归,多模态预测精度提升,边缘更清晰,计算效率更高。
  • 适合需要高精度、可解释性输出的机器人、自动驾驶等场景。

从部分观测中进行预测是世界建模的核心。现有方法通常以图像表示世界,将预测转化为随机视频生成,虽在真实感上表现优异,但像素级预测计算开销大,且难以直接用于决策。另一种方法使用视觉基础模型(VFM)特征作为世界表示,通过确定性回归预测未来状态,能直接转换为语义分割、深度等可行动信号,计算高效。然而,确定性回归会平均多个可能未来,无法捕捉不确定性,降低准确性。为此,我们提出一种生成式预测器,在VFM特征空间中进行自回归流匹配。关键洞察是:需将VFM特征编码到紧凑潜在空间以适配扩散模型。实验表明,该潜在空间比传统PCA方法更好地保留信息,适用于预测与图像生成等任务。潜在预测可轻松解码为多种有用且可解释的输出:语义分割、深度、表面法向甚至RGB。在相同架构与算力下,我们的方法在所有模态上均优于回归基线,生成结果更锐利、更准确。结果表明,对VFM特征进行随机条件生成,是未来世界模型的有前景且可扩展的基石。

原文摘要 · Abstract (English)

Forecasting from partial observations is central to world modeling. Many recent methods represent the world through images, and reduce forecasting to stochastic video generation. Although such methods excel at realism and visual fidelity, predicting pixels is computationally intensive and not directly useful in many applications, as it requires translating RGB into signals useful for decision making. An alternative approach uses features from vision foundation models (VFMs) as world representations, performing deterministic regression to predict future world states. These features can be directly translated into actionable signals such as semantic segmentation and depth, while remaining computationally efficient. However, deterministic regression averages over multiple plausible futures, undermining forecast accuracy by failing to capture uncertainty. To address this crucial limitation, we introduce a generative forecaster that performs autoregressive flow matching in VFM feature space. Our key insight is that generative modeling in this space requires encoding VFM features into a compact latent space suitable for diffusion. We show that this latent space preserves information more effectively than previously used PCA-based alternatives, both for forecasting and other applications, such as image generation. Our latent predictions can be easily decoded into multiple useful and interpretable output modalities: semantic segmentation, depth, surface normals, and even RGB. With matched architecture and compute, our method produces sharper and more accurate predictions than regression across all modalities. Our results suggest that stochastic conditional generation of VFM features offers a promising and scalable foundation for future world models.

世界建模视觉模型生成预测多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。