提升单目3D视觉定位中文本对几何单位的理解能力,显著改善定位精度。
Dual Enhancement on 3D Vision-Language Perception for Monocular 3D Visual Grounding
- 通过文本增强与几何引导双策略,强化模型对单位变换的鲁棒性
- 在远距离场景下准确率提升11.94%,达到新基准水平
- 适合研究多模态3D理解、视觉定位与语言-几何对齐的学者
单目3D视觉定位是一项新任务,旨在利用带显式几何信息的文本描述在RGB图像中定位3D物体。尽管文本包含几何细节,我们发现文本嵌入对数值大小敏感,却忽略测量单位。例如,将“米”简单映射为“分米”或“厘米”,即使物理长度不变,也会导致性能严重下降。这表明预训练语言模型对3D理解薄弱,生成误导性文本特征,阻碍3D感知。为此,我们提出两种简单有效的方法:首先引入3D文本增强(3DTE)预处理方法,通过丰富文本查询中的距离描述多样性来增强单位间映射关系的理解;其次设计文本引导几何增强(TGE)模块,将基础文本特征投影到几何一致空间,进一步增强3D文本信息,并精准引导几何特征注意力。我们在Mono3DRefer数据集上进行大量对比与消融实验,结果表明该方法显著优于先前方法,在“远距离”场景下准确率提升11.94%,达到新的最先进水平。代码将公开。
原文摘要 · Abstract (English)
Monocular 3D visual grounding is a novel task that aims to locate 3D objects in RGB images using text descriptions with explicit geometry information. Despite the inclusion of geometry details in the text, we observe that the text embeddings are sensitive to the magnitude of numerical values but largely ignore the associated measurement units. For example, simply equidistant mapping the length with unit "meter" to "decimeters" or "centimeters" leads to severe performance degradation, even though the physical length remains equivalent. This observation signifies the weak 3D comprehension of pre-trained language model, which generates misguiding text features to hinder 3D perception. Therefore, we propose to enhance the 3D perception of model on text embeddings and geometry features with two simple and effective methods. Firstly, we introduce a pre-processing method named 3D-text Enhancement (3DTE), which enhances the comprehension of mapping relationships between different units by augmenting the diversity of distance descriptors in text queries. Next, we propose a Text-Guided Geometry Enhancement (TGE) module to further enhance the 3D-text information by projecting the basic text features into geometrically consistent space. These 3D-enhanced text features are then leveraged to precisely guide the attention of geometry features. We evaluate the proposed method through extensive comparisons and ablation studies on the Mono3DRefer dataset. Experimental results demonstrate substantial improvements over previous methods, achieving new state-of-the-art results with a notable accuracy gain of 11.94\% in the "Far" scenario. Our code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。