用视觉大模型+密度建模，无监督检测自动驾驶中的异常输入。
Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving
- 用视觉大模型提取特征，再建模训练数据分布密度作为正常判断依据。
- 在4个大模型、5种密度方法上测试，显著优于现有方法。
- 无需标注异常数据，适合复杂自动驾驶场景的安全监控。
深度神经网络在自动驾驶等开放世界任务中仍面临分布偏移挑战：对未知物体（语义偏移）或光照变化（协变量偏移）的鲁棒性无法保证。因此，运行时可靠识别分布外（OOD）输入的监测机制至关重要。现有方法在自动驾驶等复杂领域未经验证，检测偏移类型有限，或需依赖有标注的异常样本。为此，我们提出一种基于原理性、无监督、模型无关的方法，统一检测语义与协变量偏移：构建训练数据特征分布的完整模型，以新样本处的密度值作为分布内（ID）得分。通过将视觉基础模型（VFMs）与多种密度建模技术结合，在4种不同主干结构的VFMs和5种密度方法上进行系统评估，首次全面对比了各类方法在多样化条件下的OOD分类能力。结果表明，基于VFM嵌入与密度估计的方法在识别分布外输入方面显著优于当前最优二分类方法。此外，该方法能有效识别可能导致下游任务出错的高风险输入，提升整体性能。总体而言，结合强健密度建模技术的视觉基础模型，有望实现复杂视觉任务中模型无关、无监督、可靠的安全部署监控。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) remain challenged by distribution shifts in complex open-world domains like automated driving (AD): Robustness against yet unknown novel objects (semantic shift) or styles like lighting conditions (covariate shift) cannot be guaranteed. Hence, reliable operation-time monitors for identification of out-of-training-data-distribution (OOD) scenarios are imperative. Current approaches for OOD classification are untested for complex domains like AD, are limited in the kinds of shifts they detect, or even require supervision with OOD samples. To prepare for unanticipated shifts, we instead establish a framework around a principled, unsupervised and model-agnostic method that unifies detection of semantic and covariate shifts: Find a full model of the training data's feature distribution, to then use its density at new points as in-distribution (ID) score. To implement this, we propose to combine Vision Foundation Models (VFMs) as feature extractors with density modeling techniques. Through a comprehensive benchmark of 4 VFMs with different backbone architectures and 5 density-modeling techniques against established baselines, we provide the first systematic evaluation of OOD classification capabilities of VFMs across diverse conditions. A comparison with state-of-the-art binary OOD classification methods reveals that VFM embeddings with density estimation outperform existing approaches in identifying OOD inputs. Additionally, we show that our method detects high-risk inputs likely to cause errors in downstream tasks, thereby improving overall performance. Overall, VFMs, when coupled with robust density modeling techniques, are promising to realize model-agnostic, unsupervised, reliable safety monitors in complex vision tasks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。