对比视觉语言与视频生成模型在空间智能上的表现,发现各有优势且可互补。
Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models

- 用轻量探针冻结模型特征,系统比较两类模型对空间智能的建模能力。
- 视觉语言模型在语义标签和实例分组上更优,视频生成模型在几何与相机运动上更强。
- 融合两类模型特征可同时提升语义与几何建模能力,适合构建更强空间智能基础模型。
空间智能需要能捕捉物理世界中语义对象与几何结构的视觉表征。目前主流的两种预训练范式是:利用语言监督将视觉观测与语义概念对齐的视觉语言模型(VLMs),以及从时序演化视觉世界中学习的视频生成模型(VGMs)。然而,哪类模型更适合空间智能仍不明确。本文首次通过冻结特征探针方法,系统比较了VLMs与VGMs在三个典型空间智能维度上的表现:语义标记、实例分组和3D几何预测。使用轻量级探针框架,实现了对两类模型冻结表征中信息编码的可控对比。实验结果表明二者存在明显互补性:VLMs在语义标记和实例分组任务上表现更优,而VGMs则为密集几何与相机运动提供了更易提取的信号。此外,简单融合两类模型特征已能在几何与语义任务上均取得优异表现,提示通过有效整合两类特征可构建更强大的空间智能基础模型。代码开源于:https://github.com/om-ai-lab/Probing-VLM-VGM。
原文摘要 · Abstract (English)
Spatial intelligence requires visual representations that capture both semantic objects and geometric structure in the physical world. To support this, two major pre-training schemes are now widely used as foundation backbones: Vision-Language Models (VLMs), which use language supervision to align visual observations with semantic concepts, and Video Generation Models (VGMs), which learn from temporally evolving visual worlds. However, it still remains unclear which pre-training scheme provides a better representation substrate for spatial intelligence. In this paper, we present the first systematic frozen-feature probing study of VLMs and VGMs across three representative axes of spatial intelligence: semantic tagging, instance grouping, and 3D geometry prediction. Using the lightweight probe, our framework enables a controlled comparison of what information is already encoded in frozen representations from two model families. Experimental results reveal a clear complementarity: VLMs are stronger at semantic tagging and instance grouping, while VGMs provide more accessible signals for dense geometry and camera motion. Moreover, a naive fusion of the two already yields a representation that excels at both geometry and semantics, suggesting a promising direction for building stronger spatial-intelligence backbones by effectively integrating features from both model families. Our code is available at \href{https://github.com/om-ai-lab/Probing-VLM-VGM}{https://github.com/om-ai-lab/Probing-VLM-VGM}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。