动态融合主体与风格LoRA,生成更协调的图像无需重训练。
Dynamic Training-Free Fusion of Subject and Style LoRAs
- 运行时按层计算KL散度,动态选择最优LoRA权重融合。
- 反向去噪阶段用CLIP/DINO评分引导隐空间修正,提升语义一致性。
- 无需训练即可适配多种主题风格组合,适合快速原型设计。
近期研究探索了多LoRA联合生成用户指定主题与风格的方法。然而,现有方法大多采用静态统计启发式融合权重,偏离了LoRA原本学习自适应特征调整的初衷,且忽视了采样输入的随机性。为此,我们提出一种全程动态、无需训练的融合框架。在前向传播中,每层应用LoRA时,动态计算基础模型原始特征与主体/风格LoRA输出特征间的KL散度,并自适应选择最合适的权重进行融合。在反向去噪阶段,进一步通过基于目标指标(如CLIP和DINO分数)的梯度修正,持续提供语义与风格指导。通过整合特征级选择与度量引导的隐空间调整两种互补机制,贯穿整个扩散过程,本方法实现了无需重训练的连贯主题-风格合成。大量实验表明,在多样主题-风格组合下,该方法在定性和定量上均优于现有最优的LoRA融合方法。
原文摘要 · Abstract (English)
Recent studies have explored the combination of multiple LoRAs to simultaneously generate user-specified subjects and styles. However, most existing approaches fuse LoRA weights using static statistical heuristics that deviate from LoRA's original purpose of learning adaptive feature adjustments and ignore the randomness of sampled inputs. To address this, we propose a dynamic training-free fusion framework that operates throughout the generation process. During the forward pass, at each LoRA-applied layer, we dynamically compute the KL divergence between the base model's original features and those produced by subject and style LoRAs, respectively, and adaptively select the most appropriate weights for fusion. In the reverse denoising stage, we further refine the generation trajectory by dynamically applying gradient-based corrections derived from objective metrics such as CLIP and DINO scores, providing continuous semantic and stylistic guidance. By integrating these two complementary mechanisms-feature-level selection and metric-guided latent adjustment-across the entire diffusion timeline, our method dynamically achieves coherent subject-style synthesis without any retraining. Extensive experiments across diverse subject-style combinations demonstrate that our approach consistently outperforms state-of-the-art LoRA fusion methods both qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。