arXiv:2510.18244cs.CV2025-10被引 1

用合成与真实数据混合训练,让3D物体识别在无标注场景下表现更优。

BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretraining

  • 先用合成数据打基础,再逐步融入真实扫描数据,渐进式适应真实环境。
  • 仅用1.5%的真实样本,零样本分类准确率提升27%,nuScenes上领先19.3%。
  • 适合自动驾驶等需要泛化能力的开放词汇3D感知任务,减少对人工标注依赖。

零样本3D物体分类对自动驾驶等实际应用至关重要,但常受合成数据与真实稀疏噪声LiDAR扫描之间显著域差距的制约。仅在合成数据上训练的方法难以泛化到户外场景,而仅在真实数据上训练则缺乏识别罕见或未见物体的语义多样性。我们提出BlendCLIP,一种多模态预训练框架,通过有策略地结合两类数据优势,弥合合成到真实的数据鸿沟。首先,我们构建了一个大规模对象级三元组数据集,包含点云、图像和文本描述,直接从真实驾驶数据中挖掘并由人工标注3D框生成。核心贡献是一种基于课程学习的数据混合策略:先在语义丰富的合成CAD数据上初始化模型,再逐步适应真实扫描特性。实验表明,该方法极具标签效率:每批次仅引入1.5%的真实样本,即可使nuScenes基准上的零样本准确率提升27%。最终模型在nuScenes和TruckScenes等复杂户外数据集上达到当前最优性能,相比最佳先前方法在nuScenes上提升19.3%,同时保持在多样化合成基准上的强泛化能力。结果表明,有效的域适应比全面真实标注更能推动鲁棒的开放词汇3D感知。代码与数据集将在接受后发布于https://github.com/kesu1/BlendCLIP。

原文摘要 · Abstract (English)

Zero-shot 3D object classification is crucial for real-world applications like autonomous driving, however it is often hindered by a significant domain gap between the synthetic data used for training and the sparse, noisy LiDAR scans encountered in the real-world. Current methods trained solely on synthetic data fail to generalize to outdoor scenes, while those trained only on real data lack the semantic diversity to recognize rare or unseen objects. We introduce BlendCLIP, a multimodal pretraining framework that bridges this synthetic-to-real gap by strategically combining the strengths of both domains. We first propose a pipeline to generate a large-scale dataset of object-level triplets -- consisting of a point cloud, image, and text description -- mined directly from real-world driving data and human annotated 3D boxes. Our core contribution is a curriculum-based data mixing strategy that first grounds the model in the semantically rich synthetic CAD data before progressively adapting it to the specific characteristics of real-world scans. Our experiments show that our approach is highly label-efficient: introducing as few as 1.5\% real-world samples per batch into training boosts zero-shot accuracy on the nuScenes benchmark by 27\%. Consequently, our final model achieves state-of-the-art performance on challenging outdoor datasets like nuScenes and TruckScenes, improving over the best prior method by 19.3\% on nuScenes, while maintaining strong generalization on diverse synthetic benchmarks. Our findings demonstrate that effective domain adaptation, not full-scale real-world annotation, is the key to unlocking robust open-vocabulary 3D perception. Our code and dataset will be released upon acceptance on https://github.com/kesu1/BlendCLIP.

3D感知零样本域适应多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。