arXiv:2412.11815cs.CV2024-12被引 9

用检索增强实现连续图像自动着色,保持角色一致性。

ColorFlow: Retrieval-Augmented Image Sequence Colorization

  • 通过检索相关彩色参考图,无需微调即可着色
  • 双分支设计分离颜色身份提取与着色过程,提升一致性
  • 新基准测试显示性能领先,适合动漫/漫画工业应用

在动漫或漫画系列等工业场景中,自动对黑白图像序列进行着色并保持角色和物体身份一致,是极具市场需求的复杂任务。尽管基于扩散模型的大规模生成模型在视觉着色方面取得进展,但可控性与身份一致性仍存挑战,难以满足工业需求。为此,我们提出 ColorFlow,一种三阶段扩散模型框架,专为工业级图像序列着色设计。不同于需逐身份微调或显式提取身份嵌入的方法,我们提出一种新颖的检索增强着色流程,通过相关彩色参考图实现通用、鲁棒的着色。该流程采用双分支结构:一支用于颜色身份提取,另一支负责着色,充分发挥扩散模型优势。利用扩散模型中的自注意力机制实现强上下文学习与颜色身份匹配。为评估模型,我们构建了 ColorFlow-Bench,一个全面的基于参考的着色基准测试。结果表明,ColorFlow 在多个指标上优于现有模型,树立了序列图像着色新标准,有望推动艺术行业应用。代码与模型已公开于项目主页:https://zhuang2002.github.io/ColorFlow/。

原文摘要 · Abstract (English)

Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.To address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: https://zhuang2002.github.io/ColorFlow/.

图像着色扩散模型检索增强身份一致

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。