用视觉大模型嵌入检测自动驾驶中的语义异常
Vision Foundation Model Embedding-Based Semantic Anomaly Detection
- 直接在图像上比较运行时嵌入与正常场景数据库
- 结合实例分割与过滤机制，准确率接近GPT-4o
- 适合需要实时异常定位的自动驾驶系统
语义异常是常见视觉元素在上下文中不合理的组合，可能导致自主系统推理失败。本文利用先进视觉基础模型的语义先验，在图像层面实现语义异常检测。提出一种框架：将运行时图像的局部视觉嵌入与被认定为安全且性能良好的正常场景数据库进行对比。考虑两种变体：一种使用原始网格嵌入，另一种采用实例分割获取以对象为中心的表示。为进一步提升鲁棒性，引入简单过滤机制抑制误报。在CARLA模拟异常上的评估显示，结合过滤的基于实例的方法性能可媲美GPT-4o，同时提供精确的异常定位。结果表明，视觉基础模型的嵌入对自主系统实时异常检测具有潜在价值。
原文摘要 · Abstract (English)
Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。