arXiv:2507.10029cs.CVcs.LG2025-07中稿 · ICCV被引 2

通过动态选择优化策略,实现低内存高质图像生成模型个性化。

Memory-Efficient Personalization of Text-to-Image Diffusion Models via Selective Optimization Strategies

  • 根据扩散过程时步动态选择低分辨率反向传播或高分辨率零阶优化。
  • 相比传统方法内存降低87%,在边缘设备上实现高质量微调。
  • 适合资源受限场景下的隐私保护图像生成模型定制,如手机端应用。

在有限计算资源和保护用户隐私的前提下,高效实现文本到图像扩散模型的个性化至关重要。本文提出一种选择性优化框架,根据扩散过程特性自适应地在低分辨率反向传播(BP-low)与高分辨率零阶优化(ZO-high)之间切换。实验表明,BP-low能有效捕捉目标特征但因分辨率不匹配导致结构失真;而ZO-high虽内存开销极小,却收敛缓慢。本框架结合两者优势:先用BP-low进行高效个性化,再以ZO-high保持结构一致性。为提升效率,引入基于时步的概率选择函数,缓解高时步下BP-low的过拟合问题,同时增强后期ZO-high的应用效果。实验显示,该方法在保持竞争力性能的同时,内存消耗显著降低87%,支持无延迟的设备端可扩展个性化。

原文摘要 · Abstract (English)

Memory-efficient personalization is critical for adapting text-to-image diffusion models while preserving user privacy and operating within the limited computational resources of edge devices. To this end, we propose a selective optimization framework that adaptively chooses between backpropagation on low-resolution images (BP-low) and zeroth-order optimization on high-resolution images (ZO-high), guided by the characteristics of the diffusion process. As observed in our experiments, BP-low efficiently adapts the model to target-specific features, but suffers from structural distortions due to resolution mismatch. Conversely, ZO-high refines high-resolution details with minimal memory overhead but faces slow convergence when applied without prior adaptation. By complementing both methods, our framework leverages BP-low for effective personalization while using ZO-high to maintain structural consistency, achieving memory-efficient and high-quality fine-tuning. To maximize the efficacy of both BP-low and ZO-high, we introduce a timestep-aware probabilistic function that dynamically selects the appropriate optimization strategy based on diffusion timesteps. This function mitigates the overfitting from BP-low at high timesteps, where structural information is critical, while ensuring ZO-high is applied more effectively as training progresses. Experimental results demonstrate that our method achieves competitive performance while significantly reducing memory consumption, enabling scalable, high-quality on-device personalization without increasing inference latency.

扩散模型低内存个性化边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。