通过几何与语义融合提升多模态模型的空间推理能力
SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion
- 设计分层适配器融合几何与语义特征，增强视觉编码器空间感知
- 在SpatialRGPT-Bench上提升至少8.0%准确率，推理内存降低约50%
- 适合需要精准空间理解的视觉问答、3D场景分析任务
多模态大语言模型虽在图像与语言任务中取得进展，但普遍存在三维空间关系推理能力不足的问题。本文提出基于几何-语义分层融合的新型视觉编码器SpatialGeo，生成具备空间感知能力的视觉嵌入。研究发现，现有模型（如CLIP）因仅依赖实例级语义特征，导致空间模糊性。为此，通过分层适配器引入纯视觉自监督学习的几何特征，弥补缺陷。模型利用预训练LLaVA进行高效训练，并采用随机特征丢弃优化策略，避免对CLIP的过度依赖。实验表明，SpatialGeo在SpatialRGPT-Bench上提升至少8.0%准确率，推理时内存消耗减少约50%。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with approximately 50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。