用多模态大模型提升图像分布外检测,兼顾近域与远域异常。
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
- 利用多模态大模型进行多轮对话式推理,融合图文信息增强判断。
- 在Food-101上显著提升近域与远域异常检测准确率,支持ImageNet-1K扩展。
- 适合关注零样本异常检测、多模态推理的科研与工程人员。
分布外(OOD)检测是关键任务,近年来受广泛关注。CLIP的出现推动了零样本OOD检测研究,多数方法采用无训练范式,依赖大语言模型(LLM)的文本知识识别异常。然而,这些方法过度依赖文本空间知识,忽视了图像空间中分布外样本检测的固有挑战。本文提出一种新框架MM-OOD,充分利用多模态大模型(MLLM)的跨模态推理能力及多轮对话机制,以提升近域与远域OOD检测性能。针对近域任务,直接输入已知类别图像与对应文本提示,由MLLM识别潜在异常;针对远域任务,提出sketch-generate-elaborate框架:先用文本提示生成异常草图,再合成视觉异常样本,最后通过多模态提示进一步细化描述。实验表明,该方法在Food-101等主流多模态数据集上取得显著提升,并验证了在ImageNet-1K上的可扩展性。
原文摘要 · Abstract (English)
Out-of-Distribution (OOD) detection is a critical task that has garnered significant attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text prompts, then generate corresponding visual OOD samples, and finally elaborate by using multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。