arXiv:2410.06104cs.CV2024-10中稿 · PRCV2024

通过低秩残差动态优化StyleGAN卷积核,提升图像生成质量。

RefineStyle: Dynamic Convolution Refinement for StyleGAN

  • 用输入图像或领域引导生成低秩残差来修正动态卷积核。
  • 图像逆向重建时畸变小,跨域编辑质量高。
  • 适合需要高效微调的图像生成与编辑场景。

在StyleGAN中,卷积核由跨图像共享的静态参数和针对每张图像的动态调制因子 $w^+\in\mathcal{W}^+$ 共同决定。因此,$\mathcal{W}^+$ 空间常用于图像逆向重建与编辑。然而,预训练模型在生成域外图像时表现受限,因 $\mathcal{W}^+$ 及其生成的卷积核能力有限,通常需全量微调或通过复杂超网络进行适配。本文提出一种高效的动态卷积核精炼策略:通过从输入图像或领域引导中学习的低秩残差来修改卷积核。这些残差由两组相同数量的令牌经矩阵乘法生成,从而控制复杂度。我们在图像逆向重建与领域适配任务中验证该方案:前者采用分组Transformer块,通过单阶段或双阶段训练学习令牌集;后者直接优化令牌集以支持目标域合成,同时保留原始内容。大量实验表明,该方法在图像逆向重建中保持低畸变,在域外编辑中实现高质量生成。

原文摘要 · Abstract (English)

In StyleGAN, convolution kernels are shaped by both static parameters shared across images and dynamic modulation factors $w^+\in\mathcal{W}^+$ specific to each image. Therefore, $\mathcal{W}^+$ space is often used for image inversion and editing. However, pre-trained model struggles with synthesizing out-of-domain images due to the limited capabilities of $\mathcal{W}^+$ and its resultant kernels, necessitating full fine-tuning or adaptation through a complex hypernetwork. This paper proposes an efficient refining strategy for dynamic kernels. The key idea is to modify kernels by low-rank residuals, learned from input image or domain guidance. These residuals are generated by matrix multiplication between two sets of tokens with the same number, which controls the complexity. We validate the refining scheme in image inversion and domain adaptation. In the former task, we design grouped transformer blocks to learn these token sets by one- or two-stage training. In the latter task, token sets are directly optimized to support synthesis in the target domain while preserving original content. Extensive experiments show that our method achieves low distortions for image inversion and high quality for out-of-domain editing.

StyleGAN动态卷积图像生成低秩优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。