通过多视图图文对齐,提升3D物体在未知类别上的识别能力。
HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition

- 用多个图像和文本描述对齐点云,构建更完整的3D理解。
- 提出解耦多正样本对比损失,在复杂负样本上表现更优。
- 轻量级文本适配器增强网络对网络文本的利用,适合实时应用。
开放集3D识别要求模型能泛化到罕见或未见类别。现有方法通常将语言-视觉知识蒸馏到3D编码器中,依赖大型2D ViT,并将每个点云与单张图像或一段文本对齐,导致表征受限于局部视角。本文提出将每个点云与多张图像和多个文本描述对齐,以获取更全面的3D对象理解。为此,设计了一种新损失函数——解耦多正样本对比损失,能同时对齐3D实例与多个匹配信号(多视图图像与多段文本),并分离正样本聚合与负样本竞争。该设计增强了对困难负样本的关注,避免了多个正样本共享同一softmax时的“聚光灯拥挤”问题。此外,引入仅作用于网络抓取句子的轻量级文本适配器,缩小与标注数据的领域差距,有效利用大规模无监督文本。模型在长尾基准上达到当前最佳的开放词汇性能,实现显著的零样本提升,同时保持高帧率。
原文摘要 · Abstract (English)
Open-set 3D recognition requires models that generalize to rare or unseen categories. Recent approaches address this by distilling language-vision knowledge into 3D encoders, typically relying on heavy 2D ViTs and aligning each point cloud with a single image or caption, thus anchoring representations to partial views. We propose aligning each point cloud with multiple images and textual descriptions to capture a more holistic understanding of 3D objects. To realize this idea, it is essential to design a loss function capable of jointly aligning a 3D instance with multiple matched signals, multi-view images and multiple texts, while separating positive aggregation from negative competition. We introduce such a function, termed the decoupled multi-positive contrastive loss. Our formulation enhances the loss's hardness-aware focus on challenging negatives, avoiding the "spotlight crowding" that occurs when many positives share the same softmax with all the negatives. Complementing this, we present a lightweight text adapter applied only to web captions, reducing the domain gap to curated annotations and enabling effective use of large-scale unsupervised text. Our model demonstrates state-of-the-art open-vocabulary performance on long-tail benchmarks, yielding substantial zero-shot improvements while sustaining high frame rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。