arXiv:2409.12960cs.CVcs.GR2024-09中稿 · SIGGRAPH被引 35

用扩散模型实现参考帧驱动的线稿动画上色,时序更连贯。

LVCD: Reference-based Lineart Video Colorization with Diffusion Models

论文配图:LVCD: Reference-based Lineart Video Colorization with Diffusion Models
图 1 · 摘自论文原文
  • 引入草图控制网络,让图像到视频扩散模型能根据线稿生成可控动画。
  • 提出参考注意力机制,有效将参考帧颜色迁移到大运动场景中。
  • 设计分段采样新方案,突破视频长度限制,可生成长时一致的上色视频。

我们提出了首个用于参考帧驱动的线稿动画上色的视频扩散框架。与以往仅依赖图像生成模型逐帧上色的方法不同,本方法利用大规模预训练视频扩散模型生成动画视频,显著提升时序一致性并更好处理大运动。首先,我们引入草图引导的ControlNet,为图像到视频扩散模型提供额外控制,实现基于线稿的可控视频生成。随后提出参考注意力机制,促进参考帧颜色向包含快速大范围运动的其他帧迁移。最后,设计一种新颖的序列采样方案,结合重叠融合模块与前参考注意力,突破视频扩散模型原有固定长度限制,实现长视频上色。定性和定量实验表明,本方法在帧质量、视频质量和时序一致性方面均显著优于当前最优技术。此外,本方法可生成高质量、长时一致且支持大运动的动画上色视频,这是此前工作无法实现的。代码与模型已公开于 https://luckyhzt.github.io/lvcd。

原文摘要 · Abstract (English)

We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at https://luckyhzt.github.io/lvcd.

视频生成扩散模型线稿上色时序一致

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。