让LoRA的每层自适应调整秩，更省内存还更精准。
Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation
- 每层秩在训练中动态调整，按需分配资源。
- 29个主体测试中，性能接近高秩LoRA但内存更低。
- 适合需要个性化图像生成且资源受限的场景。
低秩适应（LoRA）是基于预训练扩散模型生成个性化图像的主流微调方法。选择合适的秩至关重要，因为它在性能与内存消耗间权衡，但目前常依赖社区共识，未考虑个性化主体的复杂度。原因在于：为每个LoRA组件选择最佳秩的计算成本呈组合爆炸，因此普遍采用统一秩的简化方案。本文首次提出突破此限制的方法：受变分方法启发，让各层秩在针对特定主体微调时自由自适应。通过施加秩位置的重要性顺序，仅在必要时提升秩，实现高效优化。定性与定量实验表明，所提方法LoRA²在29个主体上，于DINO、CLIP-I和CLIP-T指标表现优异，同时所需内存和秩远低于高秩版本。代码已开源。
原文摘要 · Abstract (English)
Low Rank Adaptation (LoRA) is the de facto fine-tuning strategy to generate personalized images from pre-trained diffusion models. Choosing a good rank is extremely critical, since it trades off performance and memory consumption, but today the decision is often left to the community's consensus, regardless of the personalized subject's complexity. The reason is evident: the cost of selecting a good rank for each LoRA component is combinatorial, so we opt for practical shortcuts such as fixing the same rank for all components. In this paper, we take a first step to overcome this challenge. Inspired by variational methods that learn an adaptive width of neural networks, we let the ranks of each layer freely adapt during fine-tuning on a subject. We achieve it by imposing an ordering of importance on the rank's positions, effectively encouraging the creation of higher ranks when strictly needed. Qualitatively and quantitatively, our approach, LoRA$^2$, achieves a competitive trade-off between DINO, CLIP-I, and CLIP-T across 29 subjects while requiring much less memory and lower rank than high rank LoRA versions. Code: https://github.com/donaldssh/NotAllLayersAreCreatedEqual.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。