用结构化推理提升视觉语言模型的异常检测能力，实现高精度、可复现的自动驾驶场景异常识别。
Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning
- 将异常检测转化为分层语义一致性验证，通过两阶段流程增强VLM推理能力
- 相比传统提示方法，召回率提升18.5%，并自动标注约1万张真实图像
- 适配开源模型，支持本地部署，适合自动驾驶系统数据稀缺场景
自动驾驶系统仍面临长尾稀有、分布外的语义异常威胁。尽管视觉语言模型（VLM）在感知中展现出潜力，其在异常检测中的应用仍局限于调用专有模型，限制了可靠性、可复现性与部署可行性。为此，我们提出SAVANT（语义异常验证/分析工具包），一种无需依赖特定模型的结构化推理框架，将异常检测重新定义为多层语义一致性验证。通过“结构化场景描述提取”与“多模态评估”两阶段流程，现有VLM在输入图像中检测异常驾驶场景的能力显著提升。该方法取代随意提示，以语义感知方式将检测分解为四个语义域的严谨分析。我们在一组平衡的真实驾驶场景上验证，应用SAVANT使VLM绝对召回率相较提示基线提升约18.5%。此外，该增益支持可靠的大规模自动标注：利用框架内最优专有模型，我们高置信度自动标注了约10,000张真实世界图像。基于此高质量数据集，我们微调了一个7B参数的开源模型（Qwen2.5-VL），实现单次提示异常检测，达到90.8%召回率与93.8%准确率，超越所有评测模型，且支持近零成本本地部署。通过结合结构化语义推理与可扩展数据构建，我们为自动驾驶系统的语义异常检测提供了实用解决方案。
原文摘要 · Abstract (English)
Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution semantic anomalies. While VLMs have emerged as promising tools for perception, their application in anomaly detection remains largely restricted to prompting proprietary models - limiting reliability, reproducibility, and deployment feasibility. To address this gap, we introduce SAVANT (Semantic Anomaly Verification/Analysis Toolkit), a novel model-agnostic reasoning framework that reformulates anomaly detection as a layered semantic consistency verification. By applying SAVANT's two-phase pipeline - structured scene description extraction and multi-modal evaluation - existing VLMs improve their scores in detecting anomalous driving scenarios from input images. Our approach replaces ad hoc prompting with semantic-aware reasoning, transforming VLM-based detection into a principled decomposition across four semantic domains. We show that across a balanced set of real-world driving scenarios, applying SAVANT improves VLM's absolute recall by approximately 18.5% compared to prompting baselines. Moreover, this gain enables reliable large-scale annotation: leveraging the best proprietary model within our framework, we automatically labeled around 10,000 real-world images with high confidence. We use the resulting high-quality dataset to fine-tune a 7B open-source model (Qwen2.5-VL) to perform single-shot anomaly detection, achieving 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By coupling structured semantic reasoning with scalable data curation, we provide a practical solution to data scarcity in semantic anomaly detection for autonomous systems. Supplementary material: https://TUM-AVS.github.io/SAVANT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。