让手机端扩散模型个性化生成成为可能,仅更新数百参数即可适配新用户。
SPIRE: Conditional Personalization for Federated Diffusion Generative Models
- 将扩散模型拆分为共享主干和轻量客户端嵌入,实现高效个性化
- 仅更新0.01%参数即达优秀生成效果,未见客户时性能远超基线
- 理论证明梯度下降可恢复最优混合权重,适合移动端联邦学习
扩散模型虽推动生成式AI发展,但其庞大参数使设备端个性化与有效联邦学习难以实现。本文提出SPIRE框架,将客户端扩散生成转化为联邦学习中的条件生成。该方法将网络分解为(i)高容量全局主干,学习群体级得分函数;(ii)轻量可学习的客户端嵌入,编码本地数据统计。这种分离实现参数高效的微调,仅触及≤0.01%的权重。我们首次建立条件扩散训练与高斯混合模型最大似然估计之间的理论桥梁。对于两成分混合模型,证明在DDPM上对混合权重损失进行梯度下降可恢复最优混合权重,并具有无维度误差界。分析还提示客户端嵌入作为偏置,引导共享得分网络向个性化分布靠拢。实验表明,SPIRE在协作预训练中表现媲美强基线,适应未知客户端时显著超越基线,仅更新数百参数即降低核入学距离。此外,SPIRE有效缓解灾难性遗忘,对微调学习率和轮次选择具有鲁棒性。
原文摘要 · Abstract (English)
Recent advances in diffusion models have revolutionized generative AI, but their sheer size makes on device personalization, and thus effective federated learning (FL), infeasible. We propose Shared Backbone Personal Identity Representation Embeddings (SPIRE), a framework that casts per client diffusion based generation as conditional generation in FL. SPIRE factorizes the network into (i) a high capacity global backbone that learns a population level score function and (ii) lightweight, learnable client embeddings that encode local data statistics. This separation enables parameter efficient finetuning that touches $\leq 0.01\%$ of weights. We provide the first theoretical bridge between conditional diffusion training and maximum likelihood estimation in Gaussian mixture models. For a two component mixture we prove that gradient descent on the DDPM with respect to mixing weights loss recovers the optimal mixing weights and enjoys dimension free error bounds. Our analysis also hints at how client embeddings act as biases that steer a shared score network toward personalized distributions. Empirically, SPIRE matches or surpasses strong baselines during collaborative pretraining, and vastly outperforms them when adapting to unseen clients, reducing Kernel Inception Distance while updating only hundreds of parameters. SPIRE further mitigates catastrophic forgetting and remains robust across finetuning learning rate and epoch choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。