用2D模型监督3D模型，让3D视觉语言定位性能大幅提升。
From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs
- 通过可微渲染将3D点云转为2D图像，借用2D模型标签训练3D模型。
- 在开放词汇实例分割上达到25.7% mAP，比之前最优提升5.5个百分点。
- 无需3D标注，适合想用少量3D数据做高效训练的研究者。
3D视觉语言定位面临根本性数据瓶颈：2D模型训练使用数十亿图像，而3D模型仅能访问数千个标注场景，差距达六数量级，严重制约性能。我们提出LIFT-GS，一种实用的蒸馏技术，利用可微渲染桥接3D与2D监督。LIFT-GS从点云预测3D高斯表示，并将其渲染为语言条件下的2D视图掩码，从而无需任何3D标注即可使用2D基础模型（SAM、CLIP、LLaMA）进行监督。该渲染监督范式支持端到端训练完整编码器-解码器架构，且完全模型无关。LIFT-GS在开放词汇实例分割上取得25.7% mAP，优于此前最优结果（20.2%），并在指代定位任务上实现10%-30%一致提升。显著的是，预训练有效使微调数据集扩大2倍，表明3D VLG当前处于严重数据匮乏状态。
原文摘要 · Abstract (English)
3D vision-language grounding faces a fundamental data bottleneck: while 2D models train on billions of images, 3D models have access to only thousands of labeled scenes--a six-order-of-magnitude gap that severely limits performance. We introduce $\textbf{LIFT-GS}$, a practical distillation technique that overcomes this limitation by using differentiable rendering to bridge 3D and 2D supervision. LIFT-GS predicts 3D Gaussian representations from point clouds and uses them to render predicted language-conditioned 3D masks into 2D views, enabling supervision from 2D foundation models (SAM, CLIP, LLaMA) without requiring any 3D annotations. This render-supervised formulation enables end-to-end training of complete encoder-decoder architectures and is inherently model-agnostic. LIFT-GS achieves state-of-the-art results with $25.7\%$ mAP on open-vocabulary instance segmentation (vs. $20.2\%$ prior SOTA) and consistent $10-30\%$ improvements on referential grounding tasks. Remarkably, pretraining effectively multiplies fine-tuning datasets by 2X, demonstrating strong scaling properties that suggest 3D VLG currently operates in a severely data-scarce regime. Project page: https://liftgs.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。