用文本引导的动态融合方法，让视觉语言模型更轻更快地处理多视角驾驶推理。
TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning
- 根据输入问题语义，动态排序并融合多视角视觉特征。
- 在DriveLM上达到BLEU-4 56.82、计算量降低90%。
- 适合需要实时推理的自动驾驶系统部署。
视觉语言模型（VLM）在提升自动驾驶中的场景感知、推理与决策能力方面展现出巨大潜力，通过多模态融合实现。然而，现有模型存在计算开销大、多视角传感器数据整合效率低的问题，难以满足安全关键型自动驾驶系统的实时部署需求。为此，本文提出一种轻量级VLM——TS-VLM，引入新型文本引导软排序池化（TGSSP）模块。该模块基于输入查询语义对多视角视觉特征进行排序与融合，实现无需昂贵注意力机制的动态、查询感知多视图聚合。此设计确保了语义相关视图的自适应优先级，提升了多视角推理的上下文准确性。在DriveLM基准上的大量实验表明，TS-VLM在性能上超越当前最优模型：BLEU-4达56.82，METEOR为41.91，ROUGE-L为74.64，CIDEr为3.39；同时计算成本降低高达90%，最小版本仅含2010万参数，显著提升其在自动驾驶车辆中实时部署的可行性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have shown remarkable potential in advancing autonomous driving by leveraging multi-modal fusion in order to enhance scene perception, reasoning, and decision-making. Despite their potential, existing models suffer from computational overhead and inefficient integration of multi-view sensor data that make them impractical for real-time deployment in safety-critical autonomous driving applications. To address these shortcomings, this paper is devoted to designing a lightweight VLM called TS-VLM, which incorporates a novel Text-Guided SoftSort Pooling (TGSSP) module. By resorting to semantics of the input queries, TGSSP ranks and fuses visual features from multiple views, enabling dynamic and query-aware multi-view aggregation without reliance on costly attention mechanisms. This design ensures the query-adaptive prioritization of semantically related views, which leads to improved contextual accuracy in multi-view reasoning for autonomous driving. Extensive evaluations on the DriveLM benchmark demonstrate that, on the one hand, TS-VLM outperforms state-of-the-art models with a BLEU-4 score of 56.82, METEOR of 41.91, ROUGE-L of 74.64, and CIDEr of 3.39. On the other hand, TS-VLM reduces computational cost by up to 90%, where the smallest version contains only 20.1 million parameters, making it more practical for real-time deployment in autonomous vehicles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。