让手机也能个性化生成图片,内存占用降到最低。
Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
- 通过临时移除U-Net部分深层结构,实现高效内存优化。
- 训练内存降至与推理相当水平,支持设备端微调。
- 适配资源受限设备,适合移动端图像个性化应用。
近期文本到图像扩散模型的进展使得基于文本提示生成定制化图像成为可能。本文提出一种高效的基于LoRA的设备端个性化方法,可在资源受限设备上使用用户数据对预训练扩散模型进行微调。所提方法称为Hollowed Net,通过修改扩散U-Net架构,临时移除部分深层网络层,形成空心结构,从而显著降低微调时的显存占用。该方法直接应对设备端内存限制,相比以往仅关注减少训练步数或更新参数量的方法更具优势。此外,个性化后的Hollowed Net可还原回原始U-Net结构,推理无需额外内存开销。定量与定性分析表明,本方法不仅将训练显存降至与推理相近水平,且在个性化效果上保持或优于现有方法。
原文摘要 · Abstract (English)
Recent advancements in text-to-image diffusion models have enabled the personalization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove a fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training, in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without additional memory overhead. Quantitative and qualitative analyses demonstrate that our approach not only reduces training memory to levels as low as those required for inference but also maintains or improves personalization performance compared to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。