AutoLoRA提升LoRA微调模型生成多样性与质量
AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Models
- 结合自动引导与低秩适配，平衡领域一致性与生成多样性
- 在多个微调域上优于现有引导方法，生成样本更丰富高质量
- 适合需要高效定制化图像生成的开发者和研究者
低秩适配（LoRA）是一种可应用于条件生成扩散模型的微调技术，能用少量上下文样例将模型适配到特定领域、人物、风格或概念。然而，由于训练数据有限，微调后模型常表现出强烈的上下文偏差，且生成图像的多样性较低。为此，我们提出AutoLoRA，一种针对LoRA微调扩散模型的新引导技术。受其他引导方法启发，AutoLoRA在LoRA权重所代表的领域一致性与基模型的样本多样性之间寻找平衡。此外，我们证明对LoRA微调模型和基模型同时引入无分类器引导，可生成更具多样性和更高质量的样本。在多个微调的LoRA领域上的实验结果表明，该方法在选定指标上优于现有引导技术。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a fine-tuning technique that can be applied to conditional generative diffusion models. LoRA utilizes a small number of context examples to adapt the model to a specific domain, character, style, or concept. However, due to the limited data utilized during training, the fine-tuned model performance is often characterized by strong context bias and a low degree of variability in the generated images. To solve this issue, we introduce AutoLoRA, a novel guidance technique for diffusion models fine-tuned with the LoRA approach. Inspired by other guidance techniques, AutoLoRA searches for a trade-off between consistency in the domain represented by LoRA weights and sample diversity from the base conditional diffusion model. Moreover, we show that incorporating classifier-free guidance for both LoRA fine-tuned and base models leads to generating samples with higher diversity and better quality. The experimental results for several fine-tuned LoRA domains show superiority over existing guidance techniques on selected metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。