用预训练扩散模型直接生成边缘图,无需额外设计网络。
Generative Edge Detection with Stable Diffusion
- 基于预训练扩散模型的U-Net微调,直接预测潜在空间边缘图。
- 在BSDS数据集上OIS达0.880,性能媲美专用模型。
- 通过粒度控制实现多样且可控的边缘生成,适合图像生成场景。
边缘检测通常被视为像素级分类问题,主要由判别式方法解决。最近,基于扩散模型的生成式边缘检测方法开始兴起。尽管潜力巨大,但特定模块的重新训练和多步去噪推理限制了其广泛应用。我们推测原因在于未充分挖掘大规模预训练模型(如稳定扩散模型)中编码的丰富判别信息。为此,我们提出一种新方法——生成式边缘检测器(GED),充分利用预训练稳定扩散模型的潜力。该模型无需专门网络设计即可高效训练与推理,得益于预训练模型提供的高层与低层先验知识。具体而言,我们通过将潜在图像特征图作为输入,微调去噪U-Net并直接预测潜在空间边缘图。此外,由于边缘具有主观性和模糊性,我们在去噪U-Net中引入边缘粒度作为条件,以实现可控且多样的预测。进一步地,我们设计了粒度正则化,确保多个预测间的相对粒度关系。在多个数据集上进行了广泛实验,在BSDS测试集上取得了0.870和0.880的ODS与OIS指标,表现具有竞争力。
原文摘要 · Abstract (English)
Edge detection is typically viewed as a pixel-level classification problem mainly addressed by discriminative methods. Recently, generative edge detection methods, especially diffusion model based solutions, are initialized in the edge detection task. Despite great potential, the retraining of task-specific designed modules and multi-step denoising inference limits their broader applications. Upon closer investigation, we speculate that part of the reason is the under-exploration of the rich discriminative information encoded in extensively pre-trained large models (\eg, stable diffusion models). Thus motivated, we propose a novel approach, named Generative Edge Detector (GED), by fully utilizing the potential of the pre-trained stable diffusion model. Our model can be trained and inferred efficiently without specific network design due to the rich high-level and low-level prior knowledge empowered by the pre-trained stable diffusion. Specifically, we propose to finetune the denoising U-Net and predict latent edge maps directly, by taking the latent image feature maps as input. Additionally, due to the subjectivity and ambiguity of the edges, we also incorporate the granularity of the edges into the denoising U-Net model as one of the conditions to achieve controllable and diverse predictions. Furthermore, we devise a granularity regularization to ensure the relative granularity relationship of the multiple predictions. We conduct extensive experiments on multiple datasets and achieve competitive performance (\eg, 0.870 and 0.880 in terms of ODS and OIS on the BSDS test dataset).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。