自动检索LoRA并动态融合,让文生图模型更灵活高效
AutoLoRA: Automatic LoRA Retrieval and Fine-Grained Gated Fusion for Text-to-Image Generation
- 通过语义匹配自动找合适的LoRA,不依赖原始训练数据
- 按生成过程分层分时动态加权融合多个LoRA,效果更好
- 适合想快速定制文生图模型的开发者和研究者
尽管基于FLUX和Stable Diffusion v3等大规模模型在生成逼真图像方面取得进展,但其参数微调的固有困难限制了实际部署。虽然低秩适应(LoRA)已在少量参数开销下实现模型定制,但分布式开源LoRA模块的有效使用面临三大挑战:元数据标注稀疏、零样本适配需求以及多LoRA融合策略不佳。为此,我们提出新框架,实现语义驱动的LoRA检索与动态聚合,包含两个核心组件:(1) 基于权重编码的LoRA检索器,在LoRA参数矩阵与文本提示间建立共享语义空间,消除对原始训练数据的依赖;(2) 细粒度门控融合机制,跨网络层与扩散时间步计算上下文相关的融合权重,以最优方式整合多个LoRA模块。该方法显著提升图像生成性能,推动基础模型的可扩展、数据高效增强。本工作建立了社区开发的分散式LoRA与实际部署需求之间的关键桥梁,支持通过标准化适配器集成实现协作模型演进。
原文摘要 · Abstract (English)
Despite recent advances in photorealistic image generation through large-scale models like FLUX and Stable Diffusion v3, the practical deployment of these architectures remains constrained by their inherent intractability to parameter fine-tuning. While low-rank adaptation (LoRA) have demonstrated efficacy in enabling model customization with minimal parameter overhead, the effective utilization of distributed open-source LoRA modules faces three critical challenges: sparse metadata annotation, the requirement for zero-shot adaptation capabilities, and suboptimal fusion strategies for multi-LoRA fusion strategies. To address these limitations, we introduce a novel framework that enables semantic-driven LoRA retrieval and dynamic aggregation through two key components: (1) weight encoding-base LoRA retriever that establishes a shared semantic space between LoRA parameter matrices and text prompts, eliminating dependence on original training data, and (2) fine-grained gated fusion mechanism that computes context-specific fusion weights across network layers and diffusion timesteps to optimally integrate multiple LoRA modules during generation. Our approach achieves significant improvement in image generation perfermance, thereby facilitating scalable and data-efficient enhancement of foundational models. This work establishes a critical bridge between the fragmented landscape of community-developed LoRAs and practical deployment requirements, enabling collaborative model evolution through standardized adapter integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。