无需训练数据，让扩散模型快速适配新基座
Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models
- 通过投影将低秩调整从源模型迁移到目标模型
- 在多个文本到图像模型上实现零样本迁移，性能相当
- 适合资源受限场景下快速部署新模型
我们提出ProLoRA，实现文本到图像扩散模型中参数高效微调的零样本适应。ProLoRA将预训练的低秩调整（如LoRA）从源模型无须额外训练数据地迁移至目标模型。该方法克服了传统方法在更换基座模型时需重新训练的局限性，尤其在数据受限情况下更具优势。其核心是将源模型的调整投影到目标模型的权重空间，利用子空间与零空间的相似性，并选择性地对齐层进行操作。在多个主流文本到图像模型上的评估表明，该方法成功实现了知识迁移，在不重新训练的情况下达到相近性能。
原文摘要 · Abstract (English)
We introduce ProLoRA, enabling zero-shot adaptation of parameter-efficient fine-tuning in text-to-image diffusion models. ProLoRA transfers pre-trained low-rank adjustments (e.g., LoRA) from a source to a target model without additional training data. This overcomes the limitations of traditional methods that require retraining when switching base models, often challenging due to data constraints. ProLoRA achieves this via projection of source adjustments into the target model's weight space, leveraging subspace and null space similarities and selectively targeting aligned layers. Evaluations on established text-to-image models demonstrate successful knowledge transfer and comparable performance without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。