用视觉引导的可学习提示,提升弱监督单目3D检测性能。
VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection
- 基于场景视觉特征动态生成带不确定性的文本提示
- 在KITTI上实现最高4.8%的平均精度提升
- 适合做单目3D检测且缺乏标注数据的研究者
单目3D目标检测通常依赖伪标签以减少对真实标注的依赖。近期研究显示,确定性语言线索可作为有效的弱监督信号,提供互补语义上下文。然而,手工设计的文本描述难以捕捉场景中个体的固有视觉多样性,限制了模型学习场景感知表示的能力。为此,我们提出视觉引导的概率提示学习(VirPro),一种可无缝集成到多种弱监督单目3D检测框架中的自适应多模态预训练范式。具体而言,我们在不同场景中生成一组可学习的、实例相关的提示,并存储于自适应提示库(APB)中。随后引入多高斯提示建模(MGPM),将场景级视觉特征融入对应文本嵌入,使文本提示能够表达视觉不确定性。接着,从融合的视觉-语言嵌入中解码出针对性的高斯分布,由此获得每个实例的统一对象级提示嵌入。采用区域级对比匹配来强制模态对齐,使同一场景中共现物体的嵌入在潜在空间中更接近,从而增强语义一致性。在KITTI基准上的大量实验表明,集成该预训练范式可持续带来显著性能提升,相比基线最高实现4.8%的平均精度增益。代码已开源:https://github.com/AustinLCP/VirPro。
原文摘要 · Abstract (English)
Monocular 3D object detection typically relies on pseudo-labeling techniques to reduce dependency on real-world annotations. Recent advances demonstrate that deterministic linguistic cues can serve as effective auxiliary weak supervision signals, providing complementary semantic context. However, hand-crafted textual descriptions struggle to capture the inherent visual diversity of individuals across scenes, limiting the model's ability to learn scene-aware representations. To address this challenge, we propose Visual-referred Probabilistic Prompt Learning (VirPro), an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks. Specifically, we generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB). Subsequently, we introduce Multi-Gaussian Prompt Modeling (MGPM), which incorporates scene-based visual features into the corresponding textual embeddings, allowing the text prompts to express visual uncertainties. Then, from the fused vision-language embeddings, we decode a prompt-targeted Gaussian, from which we derive a unified object-level prompt embedding for each instance. RoI-level contrastive matching is employed to enforce modality alignment, bringing embeddings of co-occurring objects within the same scene closer in the latent space, thus enhancing semantic coherence. Extensive experiments on the KITTI benchmark demonstrate that integrating our pretraining paradigm consistently yields substantial performance gains, achieving up to a 4.8% average precision improvement than the baseline. Code is available at https://github.com/AustinLCP/VirPro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。