用视觉语言模型提升自动驾驶的时序理解能力，解决动态环境下的预测与决策难题。
From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
- 引入场景级思维链（Scene-CoT）和时间认知地图（TCogMap）增强时序推理
- 在TAD基准上，模型平均准确率提升17.72%，长时序任务提升10.35%
- 适合关注自动驾驶感知与推理的科研与工程人员
视觉语言模型（VLMs）正被广泛用于野外自主智能体的感知与推理核心，自动驾驶（AD）是其中最安全敏感的应用之一。可靠的时序理解对智能体预判事件、归因因果并安全行动至关重要，但现有最先进的VLMs仍难以胜任。以往视频评测聚焦于体育、烹饪等场景，缺乏专用于短/长时序自动驾驶视频的基准。为此，我们提出自动驾驶时序理解基准（TAD），包含近6000个跨7个任务的问答对，并评估9个开源与闭源通用及专用模型。当前最优模型在TAD上的表现显著低于人类水平。为提升基于VLM的驾驶智能体的时序推理能力，我们提出两种无需训练的新方法：Scene-CoT采用思维链（CoT）推理，TCogMap通过轨迹分析模块生成以自车为中心的时序认知地图，作为环绕VLM的代理工具。集成至现有VLM后，方法在TAD上平均准确率提升最高达17.72%，在STSBench上提升最高达10.35%。本工作通过引入TAD、评测顶尖模型并提出有效增强方案，旨在推动野外运行的自主驾驶系统在时序理解方面的进展。基准与评估代码已公开于Hugging Face与GitHub。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances. Reliable temporal understanding is essential for such agents to anticipate events, attribute causes, and act safely in dynamic environments, yet this remains a significant challenge even for state-of-the-art (SoTA) VLMs. Prior video benchmarks have emphasized other content (sports, cooking, etc.), yet no existing benchmark focuses exclusively on temporal understanding for both short- and long-form AD footage. To fill this gap, we present the Temporal Understanding in Autonomous Driving (TAD) benchmark, comprising nearly 6000 question-answer (QA) pairs across 7 tasks, and evaluate 9 closed- and open-source generalist as well as AD-specialist models. Current SoTA models perform substantially below human accuracy on TAD. To improve the temporal reasoning of VLM-based driving agents, we propose two novel training-free solutions: Scene-CoT, which uses Chain-of-Thought (CoT) reasoning, and TCogMap, which incorporates an ego-centric temporal cognitive map produced by a trajectory-analysis module that operates as an agentic tool around the VLM. Integrated with existing VLMs, our methods improve average accuracy on TAD by up to $17.72\%$ and by up to $10.35\%$ on STSBench. By introducing TAD, benchmarking SoTA models, and proposing effective enhancements, this work aims to catalyze further progress on temporal understanding for agentic AD systems operating in the wild. The benchmark and evaluation code are available at ${\href{https://huggingface.co/datasets/vbdai/TAD}{\text{Hugging Face}}}$ and ${\href{https://github.com/vbdi/tad_bench}{\text{GitHub}}}$, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。