提出隐私保护机制，让用户在分布式训练中安全使用ControlNet生成图像。
Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning
- 采用无梯度回传的分布式结构，降低数据泄露风险。
- 设计新采样策略和隐私激活函数，有效防御主流攻击。
- 适合关注生成模型隐私的开发者与研究人员。
随着大型生成模型的发展，ControlNet被引入以允许用户使用自有数据微调预训练模型。一个关键问题是：如何在分布式设备上训练ControlNet的同时保障用户数据隐私？我们评估了多种分布式训练方案，发现传统联邦学习和分割学习均不适用。为此，我们提出一种新型分布式学习结构，无需服务器向客户端发送梯度。通过全面分析现有威胁，我们发现除文献中提及的两种攻击外，其余攻击在控制网络的分割学习场景下均无效。为应对这些威胁，我们利用扩散模型特性，设计了一种新的前向过程时间步采样策略。同时，提出一种隐私保护激活函数，并开发一种防止私有文本提示泄露客户端的方法，专门用于扩散模型的图像生成。实验表明，所提算法与系统显著提升了ControlNet分布式训练效率，在不牺牲图像生成质量的前提下，有效保障用户数据隐私。
原文摘要 · Abstract (English)
With the emerging trend of large generative models, ControlNet is introduced to enable users to fine-tune pre-trained models with their own data for various use cases. A natural question arises: how can we train ControlNet models while ensuring users' data privacy across distributed devices? Exploring different distributed training schemes, we find conventional federated learning and split learning unsuitable. Instead, we propose a new distributed learning structure that eliminates the need for the server to send gradients back. Through a comprehensive evaluation of existing threats, we discover that in the context of training ControlNet with split learning, most existing attacks are ineffective, except for two mentioned in previous literature. To counter these threats, we leverage the properties of diffusion models and design a new timestep sampling policy during forward processes. We further propose a privacy-preserving activation function and a method to prevent private text prompts from leaving clients, tailored for image generation with diffusion models. Our experimental results demonstrate that our algorithms and systems greatly enhance the efficiency of distributed training for ControlNet while ensuring users' data privacy without compromising image generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。