用文本和分割图精准给图像中每个物体上色,解决颜色溢出和混淆问题。
Instance-aware Image Colorization with Controllable Textual Descriptions and Segmentation Masks
- 通过像素级掩码注意力机制,防止不同物体间颜色信息互串。
- 引入实例掩码与文本引导模块,避免文本错误影响其他区域上色。
- 提出多实例采样策略,提升复杂场景下细粒度着色效果。
近年来,深度学习在图像着色领域备受关注,扩散模型的成熟进一步推动了该方向的发展。然而,现有主流着色模型仍存在颜色溢出、颜色绑定错误等问题,且无法实现实例级别的着色。本文提出一种基于扩散模型的着色方法MT-Color,实现可控制的实例感知着色。为解决颜色溢出问题,设计了像素级掩码注意力机制,通过交叉注意力融合潜在特征与灰度条件特征,并利用分割掩码构建交叉注意力掩码,阻止不同实例间的像素信息交换。同时引入实例掩码与文本引导模块,提取各实例的掩码和文本表示,通过自注意力融合潜在特征,并以实例掩码构建自注意力掩码,防止文本引导误作用于其他区域,从而缓解颜色绑定错误。此外,采用多实例采样策略,分别采样每个实例区域后融合结果。我们还基于大视觉语言模型,在现有图像数据集上构建了专门用于实例级着色任务的数据集GPT-color。定性与定量实验表明,本方法与数据集均优于先前方法与数据集。
原文摘要 · Abstract (English)
Recently, the application of deep learning in image colorization has received widespread attention. The maturation of diffusion models has further advanced the development of image colorization models. However, current mainstream image colorization models still face issues such as color bleeding and color binding errors, and cannot colorize images at the instance level. In this paper, we propose a diffusion-based colorization method MT-Color to achieve precise instance-aware colorization with use-provided guidance. To tackle color bleeding issue, we design a pixel-level mask attention mechanism that integrates latent features and conditional gray image features through cross-attention. We use segmentation masks to construct cross-attention masks, preventing pixel information from exchanging between different instances. We also introduce an instance mask and text guidance module that extracts instance masks and text representations of each instance, which are then fused with latent features through self-attention, utilizing instance masks to form self-attention masks to prevent instance texts from guiding the colorization of other areas, thus mitigating color binding errors. Furthermore, we apply a multi-instance sampling strategy, which involves sampling each instance region separately and then fusing the results. Additionally, we have created a specialized dataset for instance-level colorization tasks, GPT-color, by leveraging large visual language models on existing image datasets. Qualitative and quantitative experiments show that our model and dataset outperform previous methods and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。