统一2D与3D视觉语言理解,提升多模态模型在真实场景中的表现。
Unifying 2D and 3D Vision-Language Understanding
- 基于2D预训练模型初始化,共享语言引导的掩码解码器跨模态定位物体。
- 在多个3D视觉语言定位任务中达到当前最优性能,超越基于框的方法。
- 无需3D网格重建或真值物体建议,适合具身智能与真实环境应用。
3D视觉语言学习因大规模3D数据集稀缺而受限。本文提出UniVLG,一种统一的2D与3D视觉语言理解架构,弥合了现有以2D为中心的模型与具身系统中丰富的3D感官数据之间的差距。方法通过预训练2D模型权重初始化,并在2D与3D视觉语言数据上联合训练。提出一种跨2D与3D模态共享的语言条件掩码解码器,有效实现RGB与RGB-D图像中的物体定位,优于基于框的方法。为缩小2D与3D域间差异,引入2D到3D的提升策略,使2D数据可增强3D性能。实验表明,该模型在多个3D视觉语言定位任务中达到最先进水平,验证了将2D视觉语言进展迁移到数据受限的3D领域的潜力。同时,2D与3D联合训练在不牺牲2D能力的前提下提升了跨模态性能。通过摒弃对3D网格重建和真值物体建议的依赖,UniVLG为具身对齐的现实评估设立了新标准。代码与可视化见https://univlg.github.io。
原文摘要 · Abstract (English)
Progress in 3D vision-language learning has been hindered by the scarcity of large-scale 3D datasets. We introduce UniVLG, a unified architecture for 2D and 3D vision-language understanding that bridges the gap between existing 2D-centric models and the rich 3D sensory data available in embodied systems. Our approach initializes most model weights from pre-trained 2D models and trains on both 2D and 3D vision-language data. We propose a novel language-conditioned mask decoder shared across 2D and 3D modalities to ground objects effectively in both RGB and RGB-D images, outperforming box-based approaches. To further reduce the domain gap between 2D and 3D, we incorporate 2D-to-3D lifting strategies, enabling UniVLG to utilize 2D data to enhance 3D performance. With these innovations, our model achieves state-of-the-art performance across multiple 3D vision-language grounding tasks, demonstrating the potential of transferring advances from 2D vision-language learning to the data-constrained 3D domain. Furthermore, co-training on both 2D and 3D data enhances performance across modalities without sacrificing 2D capabilities. By removing the reliance on 3D mesh reconstruction and ground-truth object proposals, UniVLG sets a new standard for realistic, embodied-aligned evaluation. Code and additional visualizations are available at https://univlg.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。