用超网络快速生成适配图像模型的LoRA权重，实现秒级个性化
LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization
- 设计超网络直接生成特定风格的LoRA权重
- 仅需少量训练步骤即可达到与全量微调相当的生成质量
- 适合需要快速定制化图像生成的用户或应用
低秩适应（LoRA）等参数高效微调（PEFT）方法为文本到图像模型的个性化提供了低内存、低存储的解决方案。然而，这些方法在实际训练时间或收敛步数上相比全模型微调并无显著提升。尽管PEFT假设生成分布的变化可通过低秩子空间中的权重调整有效建模，但未利用常见使用场景中对特定风格或身份的聚焦。观察到目标输出通常仅占LoRA训练覆盖域的一小部分，我们提出通过引入感兴趣区域的先验来缩小搜索空间。实验表明，训练一个超网络以生成LoRA权重，可在特定领域实现媲美全量微调的质量，同时实现近似瞬时的用户输入条件响应，而传统方法需数千步训练。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) and other parameter-efficient fine-tuning (PEFT) methods provide low-memory, storage-efficient solutions for personalizing text-to-image models. However, these methods offer little to no improvement in wall-clock training time or the number of steps needed for convergence compared to full model fine-tuning. While PEFT methods assume that shifts in generated distributions (from base to fine-tuned models) can be effectively modeled through weight changes in a low-rank subspace, they fail to leverage knowledge of common use cases, which typically focus on capturing specific styles or identities. Observing that desired outputs often comprise only a small subset of the possible domain covered by LoRA training, we propose reducing the search space by incorporating a prior over regions of interest. We demonstrate that training a hypernetwork model to generate LoRA weights can achieve competitive quality for specific domains while enabling near-instantaneous conditioning on user input, in contrast to traditional training methods that require thousands of steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。