用少量数据实现多风格图像编辑,自动调优适配质量。
StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editing
- 基于质量感知的低秩适配,自动确定每层最优秩。
- 仅用极少参数,多风格编辑效果超越现有方法。
- 适合需要快速适配新风格的研究与应用者。
近年来,图像编辑受到广泛关注,但通用模型在面对新风格时表现不佳。如何仅用少量成对数据和最小参数量,有效微调通用图像编辑模型以适应新风格,是关键挑战。为此,本文提出一种新型少样本多风格编辑框架。我们构建了一个包含五种不同风格的基准数据集,并提出质量感知低秩适配(StyleQoRA)。该方法通过图像质量指标估算每个单秩分量的重要性,自动确定各层最优秩。为平衡风格特化与知识共享,设计了混合路由的专家混合(MoE)LoRA:风格专属路由防止跨风格混淆,风格共享路由捕捉共性变换模式。此外,探索了在扩散-变换器(DiT)模型中插入LoRA的最佳位置,并引入对抗学习与流匹配引导扩散训练。实验表明,本方法以显著更少的LoRA参数,优于现有最先进方法。代码与数据集已开源。
原文摘要 · Abstract (English)
In recent years, image editing has garnered growing attention. However, general image editing models often fail to produce satisfactory results when confronted with new styles. The challenge lies in how to effectively fine-tune general image editing models to new styles using only a limited amount of paired data and a minimum number of parameters. To address this issue, this paper proposes a novel few-shot multi-style editing framework. For this task, we construct a benchmark dataset that encompasses five distinct styles. Correspondingly, we propose Quality-Aware Low-Rank Adaptation for few-shot multi-style editing (StyleQoRA). Our StyleQoRA can automatically determine the optimal rank for each layer through a novel approach that estimates the importance score of each single-rank component using an image quality metric. To balance specialization and knowledge sharing, we design a Mixture-of-Experts (MoE) LoRA with hybrid routing in our StyleQoRA, consisting of style-specific routing to prevent cross-style confusion and style-shared routing to capture common transformation patterns. Additionally, we explore the optimal location to insert LoRA within the Diffusion in Transformer (DiT) model and integrate adversarial learning and flow matching to guide the diffusion training process. Experimental results demonstrate that our proposed method outperforms existing state-of-the-art approaches with significantly fewer LoRA parameters. Our code and dataset are available at https://github.com/cao-cong/FSMSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。