解决草图上色中的分布偏移问题,实现高分辨率精细控制。
Towards High-resolution and Disentangled Reference-based Sketch Colorization
- 双分支结构分别建模训练与推理数据分布
- 引入格拉姆正则化损失提升跨域一致性
- 适配动漫风格,支持参考图像精细调控
草图上色是自动化生成动画与数字插画的关键任务。以往研究将主要困难归因于训练数据与测试数据之间的语义对齐分布差异,聚焦于缓解由此产生的伪影,而非从根本上解决该问题。本文提出一种直接减小分布偏移的框架,显著提升上色质量、分辨率和可控性。设计双分支结构,分别显式建模训练过程(语义对齐分支)与推理过程(语义错位分支)的数据分布;在两个分支的特征图间施加格拉姆正则化损失,有效强化跨域分布一致性与稳定性。进一步采用动漫专用标签网络从参考图像中提取细粒度属性,并调制SDXL的条件编码器以实现精确控制,同时引入插件模块增强纹理迁移能力。定量、定性对比及用户研究均表明,本方法有效克服分布偏移挑战,在质量和可控性指标上达到当前最优水平。消融实验验证了各组件的有效性。
原文摘要 · Abstract (English)
Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。