用视觉语言模型直接做3D感知，提升自动驾驶对未知物体的识别能力。
VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
- 端到端融合视觉语言模型与3D几何信息，避免多阶段误差传播。
- 在nuScenes上实现12.8%的感知准确率提升，关键依赖语义-几何联合损失。
- 适合关注自动驾驶中开放世界感知与大模型落地的科研与工程人员。
复杂交通环境中开放集感知对自动驾驶系统构成重大挑战，尤其体现在对未见物体类别的识别上，这对保障安全至关重要。视觉语言模型（VLM）凭借其丰富的世界知识和强大的语义推理能力，为解决该问题提供了新思路。然而，现有方法通常仅利用VLM提取视觉特征，并与传统目标检测器耦合，导致多阶段误差传播，影响感知精度。为此，我们提出VLM-3D，首个在自动驾驶场景中实现视觉语言模型端到端3D几何感知的框架。VLM-3D引入低秩适配（LoRA）以极低计算开销高效适配VLM至驾驶任务，并设计联合语义-几何损失：早期采用逐标记语义损失保证稳定收敛，后期引入3D IoU损失精细化优化3D边界框预测。在nuScenes数据集上的评估表明，所提联合语义-几何损失使感知准确率提升12.8%，充分验证了方法的有效性与先进性。
原文摘要 · Abstract (English)
Open-set perception in complex traffic environments poses a critical challenge for autonomous driving systems, particularly in identifying previously unseen object categories, which is vital for ensuring safety. Visual Language Models (VLMs), with their rich world knowledge and strong semantic reasoning capabilities, offer new possibilities for addressing this task. However, existing approaches typically leverage VLMs to extract visual features and couple them with traditional object detectors, resulting in multi-stage error propagation that hinders perception accuracy. To overcome this limitation, we propose VLM-3D, the first end-to-end framework that enables VLMs to perform 3D geometric perception in autonomous driving scenarios. VLM-3D incorporates Low-Rank Adaptation (LoRA) to efficiently adapt VLMs to driving tasks with minimal computational overhead, and introduces a joint semantic-geometric loss design: token-level semantic loss is applied during early training to ensure stable convergence, while 3D IoU loss is introduced in later stages to refine the accuracy of 3D bounding box predictions. Evaluations on the nuScenes dataset demonstrate that the proposed joint semantic-geometric loss in VLM-3D leads to a 12.8% improvement in perception accuracy, fully validating the effectiveness and advancement of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。