提出两种新损失,提升3D视觉定位的精度与语义理解能力。
Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding
- 引入视觉偏移损失和语言跨度损失,分别优化空间位置与词语层级匹配。
- 在ReferIt3D上达到当前最优性能,显著提升细粒度定位准确率。
- 适合关注3D场景下多模态对齐与细粒度理解的研究者。
3D视觉定位旨在根据语言描述在3D场景中识别对应物体实例。尽管已有多种基于选择框架的架构,但所用损失函数仍较简单,多数方法仅依赖候选实例上的交叉熵损失,未能建模实例间空间关系及语言描述的细粒度词级结构。现有方法通过全局类别预测或视觉-语言对比学习来补充语言嵌入,但未根本解决上述问题。为此,本文提出两种新损失:针对每个实例到真实目标的回归向量偏移的视觉级偏移损失,以及对语言描述中目标词段预测的语言相关跨度损失。同时,新架构AsphaltNet在视觉-语言融合模块中引入自上而下的双向注意力融合块,使两个损失的监督信号能反向传播至对应分支,从而帮助网络学习上下文感知的实例嵌入与定位感知的语言嵌入。实验表明,AsphaltNet在ReferIt3D基准上取得优于当前最先进方法的性能。
原文摘要 · Abstract (English)
3D visual grounding consists of identifying the instance in a 3D scene which is referred by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework, the utilized losses are comparatively under-explored. In particular, most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances, which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings, we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition, we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block, which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。