用语言描述3D空间关系,让图像模型更懂物体位置
SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning
- 通过LLM将2D图像的3D空间信息转为语言描述注入模型
- 在ADE20K上使DINOv3的mIoU提升至59.7,增益3.8%
- 适合需要理解空间布局的视觉任务,如场景理解
尽管大规模预训练图像表示模型在各类视觉任务中表现优异,但它们主要基于2D图像数据训练,难以捕捉真实世界中物体与背景之间的3D空间关系,限制了其在下游应用中的效果。为此,我们提出SpatialBoost,一个可扩展的框架,通过注入语言表达的3D空间知识来增强现有预训练视觉编码器的空间感知能力。核心思想是将2D图像中的密集3D空间信息转化为语言表述,并通过大语言模型(LLM)将其注入视觉编码器。为此,我们采用多轮思维链(CoT)推理过程,逐步融入密集空间知识并构建分层空间理解。为验证有效性,我们将SpatialBoost应用于DINOv3等先进视觉编码器,并在需3D感知与通用视觉能力的多种基准上评估其性能提升。例如,SpatialBoost使DINOv3在ADE20K上的mIoU从55.9提升至59.7,达到当前最优水平,较原始DINOv3提升3.8%。
原文摘要 · Abstract (English)
Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。