无需标注即可在画作中检测物体，利用扩散模型知识实现零样本识别
No Annotations for Object Detection in Art through Stable Diffusion
- 用视觉语言模型生成候选类别，结合Stable Diffusion进行条件检测
- 在ArtDL 2.0和IconArt上优于已有弱监督方法，首次实现艺术图像零样本检测
- 无需微调预训练模型，适用于无标注或少标注的艺术图像分析
艺术中的目标检测对数字人文具有重要价值，可加速艺术与历史图像中物体的识别。然而，人工标注需专业知识，难度大。本文提出NADA（无标注艺术目标检测），利用扩散模型的艺术知识，在无需完整边界框标注的情况下实现画作中的目标检测。该方法支持弱监督与零样本场景，不需微调预训练组件，由基于大视觉语言模型的类别提议器与基于Stable Diffusion的类别条件检测器构成。在ArtDL 2.0和IconArt两个艺术数据集上评估，显著优于现有弱监督方法，并成为首个实现艺术图像零样本目标检测的工作。代码已开源。
原文摘要 · Abstract (English)
Object detection in art is a valuable tool for the digital humanities, as it allows for faster identification of objects in artistic and historical images compared to humans. However, annotating such images poses significant challenges due to the need for specialized domain expertise. We present NADA (no annotations for detection in art), a pipeline that leverages diffusion models' art-related knowledge for object detection in paintings without the need for full bounding box supervision. Our method, which supports both weakly-supervised and zero-shot scenarios and does not require any fine-tuning of its pretrained components, consists of a class proposer based on large vision-language models and a class-conditioned detector based on Stable Diffusion. NADA is evaluated on two artwork datasets, ArtDL 2.0 and IconArt, outperforming prior work in weakly-supervised detection, while being the first work for zero-shot object detection in art. Code is available at https://github.com/patrick-john-ramos/nada
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。