arXiv:2510.17034cs.CV2025-10

让视觉语言模型学会区分‘是什么’和‘在哪’,提升3D定位准确率。

Where, Not What: Compelling Video LLMs to Learn Geometric Causality for 3D-Grounding

  • 分离2D语义与3D几何特征,分别对应‘是什么’和‘在哪’
  • 在ScanRefer和ScanQA上定位准确率显著提升,尤其在复杂户外场景
  • 无需改动推理结构,通过训练框架抑制2D特征依赖

多模态3D定位在视觉语言模型中备受关注,旨在提升复杂环境中的空间推理能力。然而,现有模型存在严重的“2D语义偏差”,过度依赖2D图像特征进行粗略定位,忽视3D几何信息,导致融合效果不佳。本文提出一种名为W2R2(What-Where Representation Re-Forming)的新训练框架,通过解耦表征学习和针对性地抑制捷径,重塑模型内部表示。将2D特征定义为识别“是什么”的语义信标,3D特征作为定位“在哪”的空间锚点,实现无需修改推理架构的精准3D定位。关键组件包括双目标损失函数:对齐损失使用改进交叉熵监督融合预测以促进多模态协同;伪标签损失通过基于间隔的机制惩罚过度依赖2D特征的伪输出。在ScanRefer和ScanQA数据集上的实验表明,W2R2显著提升了定位精度与鲁棒性,尤其在杂乱户外场景中表现突出。

原文摘要 · Abstract (English)

Multimodal 3D grounding has garnered considerable interest in Vision-Language Models (VLMs) \cite{yin2025spatial} for advancing spatial reasoning in complex environments. However, these models suffer from a severe "2D semantic bias" that arises from over-reliance on 2D image features for coarse localization, largely disregarding 3D geometric inputs and resulting in suboptimal fusion performance. In this paper, we propose a novel training framework called What-Where Representation Re-Forming (W2R2) to tackle this issue via disentangled representation learning and targeted shortcut suppression. Our approach fundamentally reshapes the model's internal space by designating 2D features as semantic beacons for "What" identification and 3D features as spatial anchors for "Where" localization, enabling precise 3D grounding without modifying inference architecture. Key components include a dual-objective loss function with an Alignment Loss that supervises fused predictions using adapted cross-entropy for multimodal synergy, and a Pseudo-Label Loss that penalizes overly effective 2D-dominant pseudo-outputs via a margin-based mechanism. Experiments conducted on ScanRefer and ScanQA demonstrate the effectiveness of W2R2, with significant gains in localization accuracy and robustness, particularly in cluttered outdoor scenes.

3D定位视觉语言模型几何因果多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。