用3D场景图引导视觉语言预训练,让模型更懂三维世界与语言的对应关系。
3D Scene Graph Guided Vision-Language Pre-training
- 以3D场景图为桥梁,统一学习视觉与语言特征。
- 在多个下游任务上超越或持平现有方法,如3D视觉定位和问答。
- 适合做三维视觉语言理解的研究者和开发者参考。
3D视觉语言(VL)推理因能连接三维物理世界与自然语言描述而受到广泛关注。现有方法通常采用特定任务、高度专门化的范式,仅关注有限的推理子任务,且严重依赖手工设计模块和辅助损失。这凸显了对更简单、统一、通用模型的需求。本文利用3D场景图与自然语言之间的内在联系,提出一种3D场景图引导的视觉语言预训练(VLP)框架。该方法通过模态编码器、图卷积层和交叉注意力层,学习可适配多种3D VL推理任务的通用表征,无需任务特异性设计。预训练目标包括:1)场景图引导的对比学习,利用3D场景图与自然语言间的强相关性,在细粒度层面对齐3D物体与文本特征;2)掩码模态学习,利用跨模态信息重建被掩码的词语和3D物体。为避免直接重建点云,我们使用位置线索预测被掩码物体的语义类别。大量实验表明,该预训练模型在微调至多个下游任务后,在3D视觉定位、3D密集描述和3D问答等任务上的表现优于或媲美现有方法。
原文摘要 · Abstract (English)
3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions. Existing approaches typically follow task-specific, highly specialized paradigms. Therefore, these methods focus on a limited range of reasoning sub-tasks and rely heavily on the hand-crafted modules and auxiliary losses. This highlights the need for a simpler, unified and general-purpose model. In this paper, we leverage the inherent connection between 3D scene graphs and natural language, proposing a 3D scene graph-guided vision-language pre-training (VLP) framework. Our approach utilizes modality encoders, graph convolutional layers and cross-attention layers to learn universal representations that adapt to a variety of 3D VL reasoning tasks, thereby eliminating the need for task-specific designs. The pre-training objectives include: 1) Scene graph-guided contrastive learning, which leverages the strong correlation between 3D scene graphs and natural language to align 3D objects with textual features at various fine-grained levels; and 2) Masked modality learning, which uses cross-modality information to reconstruct masked words and 3D objects. Instead of directly reconstructing the 3D point clouds of masked objects, we use position clues to predict their semantic categories. Extensive experiments demonstrate that our pre-training model, when fine-tuned on several downstream tasks, achieves performance comparable to or better than existing methods in tasks such as 3D visual grounding, 3D dense captioning, and 3D question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。