arXiv:2606.13460cs.CV2026-06

用视觉语言模型审计3D占据模型,提升物体与罕见类别的识别准确率。

VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models

论文配图:VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models
图 1 · 摘自论文原文
  • 训练时用离线VLM对每个物体实例进行语义审计,生成可靠性与证据信息
  • 在nuScenes上使占据模型平均mIoU从19.06提升至20.05,罕见类别提升明显
  • 无需修改推理过程,适合已有占据模型的精准化改进

语义3D占据为自动驾驶和机器人决策提供体素化世界状态,但物体及罕见类别错误会影响自由空间判断、碰撞检测和时序状态传播。我们发现,常见的将3D体素或物体特征与图像裁剪-文本嵌入对齐的VLM策略虽提升了文本空间相似性,却未可靠提升封闭集占据mIoU。针对这一不匹配,我们提出VISA,一种面向现有占据世界模型的训练时语义审计方法。VISA对每个物理物体实例的代表性图像裁剪调用离线VLM,获取包含类别假设、可能混淆、可靠性、属性和证据的结构化审计,并沿物体轨迹传播。审计结果与匹配的3D物体体素对齐,并通过可靠性加权的分类树、属性因子和场景级审计图损失,蒸馏为语义逻辑值;推理阶段保持不变且无需VLM。在nuScenes数据集上,三次运行平均,VISA使OccWorld的mIoU从19.06提升至20.05,GaussianWorld从21.36提升至21.91;在GaussianWorld上,物体mIoU从18.18升至19.16,罕见类别mIoU从15.60升至16.79。结果表明,VLM更适合作为可靠性感知的语义审计员,而非通用的图文嵌入目标。

原文摘要 · Abstract (English)

Semantic 3D occupancy provides a voxelized world state for autonomous driving and robot decision making, but object and rare-class errors can affect free-space interpretation, collision checking, and temporal state propagation. We show that a common VLM strategy, aligning 3D voxel or object features with crop-caption embeddings, improves text-space similarity without reliably improving closed-set occupancy mIoU. Motivated by this mismatch, we propose VISA, a training-time semantic auditing approach for existing occupancy world models. VISA queries an offline VLM on a representative crop of each physical object instance, obtains a structured audit with class hypotheses, plausible confusions, reliability, attributes, and evidence, and propagates it along the object track. The audit is grounded to matched 3D object voxels and distilled into semantic logits through reliability-weighted taxonomy, attribute-factor, and scene-level audit graph losses, while inference remains unchanged and requires no VLM. On nuScenes, averaged across three runs, VISA improves OccWorld from 19.06 to 20.05 mIoU and GaussianWorld from 21.36 to 21.91 mIoU; on GaussianWorld, object mIoU improves from 18.18 to 19.16 and rare-class mIoU from 15.60 to 16.79. These results suggest that VLMs are better suited to closed-set occupancy as reliability-aware semantic auditors than as generic caption-embedding targets.

3D占据语义审计VLM自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。