用扩散模型生成特征，让识别模型更抗遮挡。
D-Feat Occlusions: Diffusion Features for Robustness to Partial Visual Occlusions in Object Recognition
- 用冻结的扩散模型提取图像补全特征，增强遮挡感知。
- 在ImageNet上对Transformer和ConvNet均提升遮挡鲁棒性。
- 适用于真实世界遮挡场景，适合做视觉健壮性研究者参考。
扩散模型在视觉任务中表现突出。本文提出一种利用冻结扩散模型的流水线，提升分类模型在物体识别任务中对遮挡的鲁棒性。将遮挡视为图像补全问题，假设被遮挡像素为‘缺失’，利用扩散特征来‘幻化’被遮挡物体的视觉特征。设计了输入级和特征级两种增强方式：输入级通过微调补全遮挡区域的图像；特征级则用中间扩散特征增强分类特征。实验表明，在ImageNet上对Transformer和ConvNets均有效提升对模拟遮挡的鲁棒性。还构建了一个包含真实遮挡的数据集，验证了该方法在现实场景下的优越性。
原文摘要 · Abstract (English)
Applications of diffusion models for visual tasks have been quite noteworthy. This paper targets making classification models more robust to occlusions for the task of object recognition by proposing a pipeline that utilizes a frozen diffusion model. Diffusion features have demonstrated success in image generation and image completion while understanding image context. Occlusion can be posed as an image completion problem by deeming the pixels of the occluder to be `missing.' We hypothesize that such features can help hallucinate object visual features behind occluding objects, and hence we propose using them to enable models to become more occlusion robust. We design experiments to include input-based augmentations as well as feature-based augmentations. Input-based augmentations involve finetuning on images where the occluder pixels are inpainted, and feature-based augmentations involve augmenting classification features with intermediate diffusion features. We demonstrate that our proposed use of diffusion-based features results in models that are more robust to partial object occlusions for both Transformers and ConvNets on ImageNet with simulated occlusions. We also propose a dataset that encompasses real-world occlusions and demonstrate that our method is more robust to partial object occlusions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。