用预训练扩散模型的注意力机制实现精准图像着色,无需微调。
Leveraging the Powerful Attention of a Pre-trained Diffusion Model for Exemplar-based Image Colorization
- 利用扩散模型自注意力捕捉输入与参考图的语义对应关系。
- 在335对数据上达到FID 95.27(画质)和SI-FID 5.51(参考保真度)。
- 无需微调,适合快速部署于图像着色任务。
示例式图像着色旨在利用参考彩色图像为灰度图像着色,确保参考颜色根据语义相似性映射到对应区域。为实现区域间精确语义匹配,本文利用大规模数据训练的预训练扩散模型中的自注意力模块,其具备强大注意力能力。提出一种无需微调的新方法,贡献如下:首先,引入双注意力引导着色迁移。通过自注意力模块计算输入与参考图像间的注意力图,有效捕获语义对应关系;随后依据该注意力图将参考图像的颜色特征转移到输入图像的语义匹配区域,并替换灰度特征。特别地,分别对灰度图和彩色图计算双注意力图,提升语义对齐精度。其次,提出无分类器着色引导,通过融合带色与不带色输出,增强转移颜色质量。实验表明,本方法在图像质量与参考保真度上均优于现有技术。在前人研究的335组输入-参考配对上,获得FID 95.27(图像质量)和SI-FID 5.51(参考一致性)。
原文摘要 · Abstract (English)
Exemplar-based image colorization aims to colorize a grayscale image using a reference color image, ensuring that reference colors are applied to corresponding input regions based on their semantic similarity. To achieve accurate semantic matching between regions, we leverage the self-attention module of a pre-trained diffusion model, which is trained on a large dataset and exhibits powerful attention capabilities. To harness this power, we propose a novel, fine-tuning-free approach based on a pre-trained diffusion model, making two key contributions. First, we introduce dual attention-guided color transfer. We utilize the self-attention module to compute an attention map between the input and reference images, effectively capturing semantic correspondences. The color features from the reference image is then transferred to the semantically matching regions of the input image, guided by this attention map, and finally, the grayscale features are replaced with the corresponding color features. Notably, we utilize dual attention to calculate attention maps separately for the grayscale and color images, achieving more precise semantic alignment. Second, we propose classifier-free colorization guidance, which enhances the transferred colors by combining color-transferred and non-color-transferred outputs. This process improves the quality of colorization. Our experimental results demonstrate that our method outperforms existing techniques in terms of image quality and fidelity to the reference. Specifically, we use 335 input-reference pairs from previous research, achieving an FID of 95.27 (image quality) and an SI-FID of 5.51 (fidelity to the reference). Our source code is available at https://github.com/satoshi-kosugi/powerful-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。