用大模型生成假异常图像,让分类器更准区分分布外数据。
FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector
- 用多模型协作生成语义相似但不同的假异常图像
- 在多个基准上实现当前最优的分布外检测效果
- 适合需要提升模型鲁棒性的实际部署场景
开放世界应用中,分布外(OOD)检测对保障机器学习模型安全至关重要。核心挑战在于缓解模型对分布外数据的过度自信。尽管已有方法通过辅助异常数据集或合成异常特征取得进展,但受限于数据采集成本或简化假设。本文提出新框架FodFoM,创新性地结合多个基础模型生成两类挑战性假异常图像用于分类器训练。第一类利用BLIP-2的图像描述能力、CLIP的视觉-语言知识与Stable Diffusion的图像生成能力,构建语义上接近但不同于分布内(ID)图像的假异常图像;第二类则借助GroundingDINO的目标检测能力,通过模糊ID图像中的前景物体生成纯背景图像。该框架可灵活融合多种现有OOD检测方法。大量实证评估表明,加入生成假图像后,分类器对真实OOD图像的识别更准确,在多个基准上达到新的最先进性能。代码已公开于https://github.com/Cverchen/ACMMM2024-FodFoM。
原文摘要 · Abstract (English)
Out-of-Distribution (OOD) detection is crucial when deploying machine learning models in open-world applications. The core challenge in OOD detection is mitigating the model's overconfidence on OOD data. While recent methods using auxiliary outlier datasets or synthesizing outlier features have shown promising OOD detection performance, they are limited due to costly data collection or simplified assumptions. In this paper, we propose a novel OOD detection framework FodFoM that innovatively combines multiple foundation models to generate two types of challenging fake outlier images for classifier training. The first type is based on BLIP-2's image captioning capability, CLIP's vision-language knowledge, and Stable Diffusion's image generation ability. Jointly utilizing these foundation models constructs fake outlier images which are semantically similar to but different from in-distribution (ID) images. For the second type, GroundingDINO's object detection ability is utilized to help construct pure background images by blurring foreground ID objects in ID images. The proposed framework can be flexibly combined with multiple existing OOD detection methods. Extensive empirical evaluations show that image classifiers with the help of constructed fake images can more accurately differentiate real OOD images from ID ones. New state-of-the-art OOD detection performance is achieved on multiple benchmarks. The code is available at \url{https://github.com/Cverchen/ACMMM2024-FodFoM}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。