用噪声嵌入聚合实现扩散模型私有化适配，效果远超传统微调。
Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings
- 通过文本反转学习图像嵌入，再对嵌入聚合加噪以保障隐私
- 在相同隐私预算下，生成图像质量接近非私有基线，而DP-SGD失效
- 适合需保护敏感数据的个性化生成场景，如艺术风格迁移
个性化大规模扩散模型会带来严重隐私风险，尤其在小规模敏感数据上微调时。常规方法使用差分隐私随机梯度下降（DP-SGD）虽能提供隐私保护，但在小数据情况下因需添加大量噪声而导致性能严重下降。本文提出一种新方法——基于文本反转的差分隐私聚合（DPAgg-TI），通过学习图像嵌入并对其聚合结果添加校准噪声，实现正式的差分隐私保证，同时保持高输出保真度。实验表明，在相同隐私预算下，DPAgg-TI 在风格适配任务中表现显著优于 DP-SGD，能成功生成单艺术家私有艺术品及巴黎2024奥运会图标风格的高质量图像，而 DP-SGD 无法生成有意义输出。
原文摘要 · Abstract (English)
Personalizing large-scale diffusion models poses serious privacy risks, especially when adapting to small, sensitive datasets. A common approach is to fine-tune the model using differentially private stochastic gradient descent (DP-SGD), but this suffers from severe utility degradation due to the high noise needed for privacy, particularly in the small data regime. We propose an alternative that leverages Textual Inversion (TI), which learns an embedding vector for an image or set of images, to enable adaptation under differential privacy (DP) constraints. Our approach, Differentially Private Aggregation via Textual Inversion (DPAgg-TI), adds calibrated noise to the aggregation of per-image embeddings to ensure formal DP guarantees while preserving high output fidelity. We show that DPAgg-TI outperforms DP-SGD finetuning in both utility and robustness under the same privacy budget, achieving results closely matching the non-private baseline on style adaptation tasks using private artwork from a single artist and Paris 2024 Olympic pictograms. In contrast, DP-SGD fails to generate meaningful outputs in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。