让视觉语言模型学会看空间关系,靠的是世界模型的几何指导。
GeoWorld-VLM: Geometry from World Models for Vision-Language Models

- 用冻结的世界模型生成多视角空间信号,引导视觉编码器对齐
- 在两个基准上提升约4%的空间推理准确率
- 适合需要增强空间理解力的视觉语言模型开发者
现代视觉语言模型(VLMs)在语义识别上表现良好,但在左右、上下、前后等基本空间关系判断上仍脆弱。问题根源在于视觉路径压缩或丢失关键3D结构信息,导致语言模型接收的图像表征已不足以支撑可靠的空间判断。我们提出GeoWorld-VLM,一种仅微调图像编码器和多模态投影器的蒸馏框架,将冻结的相机条件视频世界模型中的几何结构迁移到VLM中。给定图像、提示词和采样相机轨迹,世界模型教师将静态输入转化为合成的多视角空间信号。训练结合空间答案监督、师生特征对齐及原始VLM保留锚点。由于语言模型保持冻结,GeoWorld-VLM在保留原有语言能力的同时,将空间性能提升归因于视觉路径的增强。我们在两种不同架构的VLM上应用该方法,均在What'sUp和VSR基准上获得约4%的性能提升,表明该方法具有跨模型结构和数据集的泛化能力。
原文摘要 · Abstract (English)
Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations such as left of, on, behind, and between. One cause of this failure arises before language reasoning begins: the visual pathway may compress or discard critical 3D structural cues during feature extraction, so the language model receives image representations that are already insufficient for reliable spatial judgment. We introduce GeoWorld-VLM, a VLM-side distillation framework that transfers geometric structure from frozen camera-conditioned video world models into VLMs. GeoWorld-VLM fine-tunes only the image encoder and multimodal projector, aligning post-projector image features with intermediate world-model representations while leaving the main backbone frozen. Given images, a prompt, and a sampled camera trajectory, the world-model teacher converts static visual input into a synthetic multi-view spatial signal. Training combines spatial answer supervision, teacher-student feature alignment, and a preservation anchor to the original VLM. Since the language model remains frozen, GeoWorld-VLM preserves the original model's linguistic capabilities while attributing spatial improvements to the enhanced visual pathway. To evaluate the effectiveness and generality of the proposed method, we apply GeoWorld-VLM to two distinct VLM architectures and observe consistent improvements across both backbones. GeoWorld-VLM improves performance by approximately 4 percent on both the What'sUp and VSR benchmarks, suggesting that world-model-guided visual alignment generalizes across model structures and spatial reasoning datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。