用视觉语言模型预测驾驶属性,让汽车更好应对未知障碍。
VOLA: Improving Open-World Driving by VLM-Based Semantic Attribute Prediction

- 直接用Qwen3.5图像令牌生成语义表示,轻量级解码器输出高分辨率属性图。
- 在真实场景中对未见障碍物的脆弱性召回率达69.4%,显著优于基线。
- 无需生成文本或外部分割模型,适合开放世界自动驾驶应用。
现实驾驶是开放世界的:车辆可能遇到床垫、鹿等训练数据之外的物体。仅命名物体不够,系统需判断如何处理每个区域——能否驶过,碰撞严重程度如何?因此,我们把场景感知从类别标签转向密集的动作相关属性,每个像素标记其对行驶的影响,而非物体名称。采用两个有序属性:7级可行驶性与5级脆弱性。直接读取Qwen3.5图像令牌隐藏状态作为空间语义表示,通过轻量级边界感知解码器将其转化为高分辨率属性图。整个过程无需自回归文本生成,也无需SAM等外部掩码模型。在CARLA构建的密集属性标签上训练,并测试在真实场景和未见障碍物上的迁移能力。相比仅用视觉的分割器和提示式VLM分割器,本模型在常见类别上表现相当,且对真实开放世界异常的迁移性能显著提升:脆弱性等级召回率达69.4%,优于最佳视觉基线(57.1%)和最佳提示式VLM基线(53.9%)。结果表明,VLM图像令牌能为训练词汇外物体的驾驶属性迁移提供有效语义线索。
原文摘要 · Abstract (English)
Driving in the real world is open-world: a car may encounter a fallen mattress, a deer, or other objects outside its training data. Naming them is not enough. The system must know how to treat each region: can it drive over it, and how severe would a collision be? We therefore shift scene perception from category labels to dense action-relevant attributes, where each pixel is labeled by how it should affect motion rather than by object name. We instantiate this general formulation with two ordered attributes: 7-rank drivability and 5-rank vulnerability. We read Qwen3.5 image-token hidden states directly as a spatial semantic representation. A lightweight boundary-aware decoder then turns this coarse token grid into sharp full-resolution attribute maps. The whole process requires neither autoregressive text generation nor an external mask model such as SAM. We train on dense attribute labels built in CARLA and test transfer to real scenes and to novel obstacles never seen in training. We compare with vision-only segmenters trained on the same attributes and prompted VLM segmenters. Our model matches strong vision-only segmenters on familiar categories and improves transfer to real open-world anomalies, reaching 69.4% mean vulnerability-rank recall versus 57.1% for the best vision-only baseline and 53.9% for the best prompted VLM baseline. These results show that VLM image tokens provide useful semantic cues for transferring driving attributes to objects outside the training vocabulary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。