无需训练即可组合多个LoRA,让图像生成更灵活准确。
LoRAtorio: An intrinsic approach to LoRA Skill Composition
- 通过分析潜在空间中的噪声相似性,动态加权多个LoRA输出。
- 在多LoRA组合上提升性能,ClipScore最高提升1.3%。
- 适合需要灵活调整视觉风格的生成应用,如角色定制与艺术创作。
低秩适配(LoRA)已成为文本到图像扩散模型中个性化视觉概念(如角色、风格、物体)的常用技术。然而,现有方法在开放场景下组合多个LoRA适配器时表现不佳,尤其当所需技能数量和类型未知时。本文提出LoRAtorio,一种无需训练的多LoRA组合框架,利用模型内在行为。该方法基于两个观察:(1)在窄域上训练的LoRA会产生偏离基础模型的去噪输出;(2)当输入分布外时,LoRA输出行为更接近基础模型。这一平衡在单个LoRA时表现优异,但多LoRA加载后性能下降。我们的方法在潜在空间中将图像划分为空间块,计算每块预测噪声与基础模型的余弦相似度,构建空间感知权重矩阵,引导加权融合多LoRA输出。为缓解领域偏移,我们改进了无分类器引导,引入基础模型的无条件得分。进一步扩展至动态模块选择设置,支持推理时从大池中选取相关LoRA。LoRAtorio实现当前最优性能,剪辑得分最高提升1.3%,在GPT-4V成对评估中胜率高达72.43%,且可泛化至多个潜在扩散模型。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models, enabling the personalisation of visual concepts such as characters, styles, and objects. However, existing approaches struggle to effectively compose multiple LoRA adapters, particularly in open-ended settings where the number and nature of required skills are not known in advance. In this work, we present LoRAtorio, a novel train-free framework for multi-LoRA composition that leverages intrinsic model behaviour. Our method is motivated by two key observations: (1) LoRA adapters trained on narrow domains produce denoised outputs that diverge from the base model, and (2) when operating out-of-distribution, LoRA outputs show behaviour closer to the base model than when conditioned in distribution. The balance between these two observations allows for exceptional performance in the single LoRA scenario, which nevertheless deteriorates when multiple LoRAs are loaded. Our method operates in the latent space by dividing it into spatial patches and computing cosine similarity between each patch's predicted noise and that of the base model. These similarities are used to construct a spatially-aware weight matrix, which guides a weighted aggregation of LoRA outputs. To address domain drift, we further propose a modification to classifier-free guidance that incorporates the base model's unconditional score into the composition. We extend this formulation to a dynamic module selection setting, enabling inference-time selection of relevant LoRA adapters from a large pool. LoRAtorio achieves state-of-the-art performance, showing up to a 1.3% improvement in ClipScore and a 72.43% win rate in GPT-4V pairwise evaluations, and generalises effectively to multiple latent diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。