提出一致性信噪比噪声分配,提升私密学习的效率与效果
Rethinking Layer-wise Gaussian Noise Injection: Bridging Implicit Objectives and Privacy Budget Allocation
- 构建统一分析框架,揭示分层噪声注入的隐含优化目标
- 新策略在隐私预算不变下显著提升模型性能
- 适用于需要高隐私保障的集中式与联邦学习场景
分层高斯机制(LGM)通过向分段梯度向量注入噪声,提升差分隐私深度学习的灵活性。然而现有方法多依赖启发式噪声分配,缺乏对噪声分配与隐私-效用权衡之间理论关联的深入理解。本文提出统一分析框架,系统连接分层噪声注入策略与其隐含优化目标及隐私预算分配。分析发现,多个现有方法优化了不当目标——或忽略层间信噪比(SNR)一致性,或导致隐私预算利用效率低下。为此,我们提出一种SNR一致性噪声分配策略,兼顾两者,实现更优信号保留与隐私预算利用率。在集中式与联邦学习设置下的大量实验表明,该方法持续优于现有策略,获得更优隐私-效用权衡。本框架不仅为先前方法提供诊断视角,也为设计自适应、高效的深度模型噪声注入方案提供理论指导。
原文摘要 · Abstract (English)
Layer-wise Gaussian mechanisms (LGM) enhance flexibility in differentially private deep learning by injecting noise into partitioned gradient vectors. However, existing methods often rely on heuristic noise allocation strategies, lacking a rigorous understanding of their theoretical grounding in connecting noise allocation to formal privacy-utility tradeoffs. In this paper, we present a unified analytical framework that systematically connects layer-wise noise injection strategies with their implicit optimization objectives and associated privacy budget allocations. Our analysis reveals that several existing approaches optimize ill-posed objectives -- either ignoring inter-layer signal-to-noise ratio (SNR) consistency or leading to inefficient use of the privacy budget. In response, we propose a SNR-Consistent noise allocation strategy that unifies both aspects, yielding a noise allocation scheme that achieves better signal preservation and more efficient privacy budget utilization. Extensive experiments in both centralized and federated learning settings demonstrate that our method consistently outperforms existing allocation strategies, achieving better privacy-utility tradeoffs. Our framework not only offers diagnostic insights into prior methods but also provides theoretical guidance for designing adaptive and effective noise injection schemes in deep models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。