通过相关噪声提升隐私训练效率，显著加速大模型隐私学习。
Cocoon: A System Architecture for Differentially Private Training with Correlated Noises
- 用预计算的关联噪声减少训练中噪声干扰，提升隐私保护效率。
- 在真实FPGA系统上实现2.33-10.82倍性能提升（嵌入层优化）。
- 适合需要高隐私保障的大规模模型训练场景。
机器学习模型会记忆并泄露训练数据，对数据所有者造成严重隐私风险。差分隐私（DP）训练算法（如DP-SGD）被视为解决方案，但其在每轮训练中添加噪声，导致模型准确率下降。为改善精度，新方法引入精心设计的相关噪声，使噪声在迭代间相互抵消。我们首次系统性地分析了此类机制，发现当模型较大或使用大型嵌入表时，仍存在显著开销。基于此，我们提出Cocoon——一种软硬件协同设计框架，通过预计算并以合并格式存储相关噪声（Cocoon-Emb），加速嵌入型模型；并通过定制近内存处理设备（Cocoon-NMP）支持大模型训练。在基于FPGA的NMP原型系统上，Cocoon实现2.33-10.82倍（Cocoon-Emb）和1.55-3.06倍（Cocoon-NMP）性能提升。
原文摘要 · Abstract (English)
Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP), such as DP-SGD, have been gaining attention as a solution. However, DP-SGD adds a noise at each training iteration, which degrades the accuracy of the trained model. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms, for the first time to the best of our knowledge, and show they incur non-negligible overheads when the model is large or uses large embedding tables. Motivated by the analysis, we propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises. Cocoon accelerates models with embedding tables through pre-computing and storing correlated noises in a coalesced format (Cocoon-Emb), and supports large models through a custom near-memory processing device (Cocoon-NMP). On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 2.33-10.82x(Cocoon-Emb) and 1.55-3.06x (Cocoon-NMP).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。