无需训练即可自适应融合风格与主体的LoRA方法
Subject or Style: Adaptive and Training-Free Mixture of LoRAs
- 基于能量、风格差异和时间步动态选择风格或主体LoRA
- 在图像生成中平衡主体与风格，性能超越现有方法
- 适合需要快速部署且兼顾生成质量的用户
通过低秩适配（LoRA）微调在主体驱动或风格驱动生成任务中表现优异。已有研究尝试组合不同LoRA以联合生成风格与内容，但当前方法难以平衡主体与风格，且常需额外训练。近期的K-LoRA提出无训练的LoRA融合方法，但涉及多个超参数，难以适配所有风格与主体。本文提出EST-LoRA，一种无需训练的自适应LoRA融合方法。综合考虑矩阵能量、风格差异分数与时间步三个关键因素，类比专家混合（MoE）架构，在每个注意力层自适应地在主体LoRA与风格LoRA间选择。该集成选择机制确保生成过程中两者贡献均衡。实验表明，EST-LoRA在定性与定量评估中均优于现有先进方法，并在生成速度上快于其他高效融合方案。代码已公开：https://anonymous.4open.science/r/EST-LoRA-F318。
原文摘要 · Abstract (English)
Fine-tuning models via Low-Rank Adaptation (LoRA) demonstrates remarkable performance in subject-driven or style-driven generation tasks. Studies have explored combinations of different LoRAs to jointly generate learned styles and content. However, current methods struggle to balance the original subject and style, and often require additional training. Recently, K-LoRA proposed a training-free LoRA fusion method. But it involves multiple hyperparameters, making it difficult to adapt to all styles and subjects. In this paper, we propose EST-LoRA, a training-free adaptive LoRA fusion method. It comprehensively considers three critical factors: \underline{E}nergy of matrix, \underline{S}tyle discrepancy scores and \underline{T}ime steps. Analogous to the Mixture of Experts (MoE) architecture, the model adaptively selects between subject LoRA and style LoRA within each attention layer. This integrated selection mechanism ensures balanced contributions from both components during the generation process. Experimental results show that EST-LoRA outperforms state-of-the-art methods in both qualitative and quantitative evaluations and achieves faster generation speed compared to other efficient fusion approaches. Our code is publicly available at: https://anonymous.4open.science/r/EST-LoRA-F318.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。