改进LoRA风格迁移,让内容和风格更一致。
ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer
- 让LoRA直接预测原图而非噪声,提升一致性。
- 两阶段训练分离内容与风格学习,减少泄露。
- 支持推理时动态调节风格强度,灵活可控。
风格迁移旨在将参考图像的风格转移到目标图像的内容上。基于低秩适应(LoRA)的方法虽能有效捕捉单张图像的风格,但仍存在内容不一致、风格错位和内容泄露等问题。本文系统分析了标准扩散参数化(以预测噪声为目标)在风格迁移中的局限性。为此,提出ConsisLoRA:通过优化LoRA权重直接预测原始图像而非噪声,从而增强内容与风格一致性。采用两步训练策略,解耦参考图像中内容与风格的学习过程。为同时保留内容图像的全局结构与局部细节,引入渐进式损失过渡策略。此外,设计一种推理引导方法,实现生成过程中对内容与风格强度的连续控制。定性和定量评估表明,该方法显著提升内容与风格一致性,有效降低内容泄露。
原文摘要 · Abstract (English)
Style transfer involves transferring the style from a reference image to the content of a target image. Recent advancements in LoRA-based (Low-Rank Adaptation) methods have shown promise in effectively capturing the style of a single image. However, these approaches still face significant challenges such as content inconsistency, style misalignment, and content leakage. In this paper, we comprehensively analyze the limitations of the standard diffusion parameterization, which learns to predict noise, in the context of style transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method that enhances both content and style consistency by optimizing the LoRA weights to predict the original image rather than noise. We also propose a two-step training strategy that decouples the learning of content and style from the reference image. To effectively capture both the global structure and local details of the content image, we introduce a stepwise loss transition strategy. Additionally, we present an inference guidance method that enables continuous control over content and style strengths during inference. Through both qualitative and quantitative evaluations, our method demonstrates significant improvements in content and style consistency while effectively reducing content leakage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。