用1比特梯度压缩实现更快更私密的深度学习，无需加噪。
FERRET: Private Deep Learning Faster And Better Than DPSGD
- 基于符号梯度与伯努利掩码，仅传输每组参数1个符号位。
- 在epsilon=0.5时，比DPSGD低3倍困惑度，接近无隐私保护效果。
- 训练速度提升5倍，且数学上可证明隐私，适合高效私密训练场景。
我们从互信息差分隐私（MI-DP）视角重新审视1比特梯度压缩。在此基础上提出FERRET——一种快速且有效的伦理训练限制发布机制，对每组参数最多传输1个符号位，并采用伯努利掩码。理论上，每个触发组泄露不超过ln2纳特；经采样率s、训练步数T、触发概率p后，总隐私损失epsilon = G * T * s * p * ln2。因此，FERRET在epsilon ∈ [0.1, 2]范围内实现MI-DP，无需加性噪声。实践中，在五种LLM（137M-1.8B参数）上评估三种粒度：FERRET-MAX（最细）、FERRET-EIGHTH（中等）、FERRET-2（最粗），对比DPSGD和非私密基线。所有方法均训练1、3、5个周期。性能方面，所有设置下FERRET-MAX/EIGHTH均优于DPSGD困惑度；在epsilon=0.5、5个周期时，FERRET-EIGHTH达3.98困惑度，远超DPSGD的11.61（2.9倍优势），仅比非私密基线（3.25）高23%。隐私方面，FERRET-MAX/EIGHTH的MI-AUC维持在随机水平（~0.51），匹配DPSGD vs 非私密的0.76–0.99表现；而FERRET-2因可用空间不足，泄漏略高（~0.55）。效率方面，更严格的预算触发更少符号，使FERRET训练时间仅为DPSGD的19%-33%，非私密基线的34%-36%。结论：基于符号的MI-DP更接近隐私-效用-性能三重困境的完美解：FERRET训练最快可达5倍加速，困惑度为DPSGD的1/3，仅比非私密高1.2倍，同时提供零加噪的数学可证隐私保障。结果还表明，在某些情况下，掩码1比特更新可达到非私密训练的性能，同时保护数据。
原文摘要 · Abstract (English)
We revisit 1-bit gradient compression through the lens of mutual-information differential privacy (MI-DP). Building on signSGD, we propose FERRET--Fast and Effective Restricted Release for Ethical Training--which transmits at most one sign bit per parameter group with Bernoulli masking. Theory: We prove each fired group leaks at most ln 2 nats; after subsampling with rate s, the total privacy loss of G groups trained for T steps with firing probability p is epsilon = G * T * s * p * ln 2. Thus FERRET achieves MI-DP for epsilon in [0.1, 2] without additive noise. Practice: We evaluate three granularities--FERRET-MAX (finest), FERRET-EIGHTH (medium), and FERRET-2 (coarsest)--on five LLMs (137M-1.8B parameters) against DPSGD and Non-DP baselines. All methods trained for 1, 3, and 5 epochs. Utility: Across all settings, FERRET-MAX/EIGHTH beat DPSGD's perplexity. At epsilon=0.5, 5 epochs: FERRET-EIGHTH achieves 3.98 perplexity vs DPSGD's 11.61 (2.9x better), within 23% of Non-DP (3.25). Privacy: MI-AUC stays at chance for FERRET-MAX/EIGHTH (~0.51), matching DPSGD vs Non-DP's 0.76-0.99. FERRET-2 shows higher leakage (~0.55) due to lower headroom. Efficiency: Stricter budgets fire fewer signs, so FERRET uses 19-33% of DPSGD's training time and only 34-36% of Non-DP training time. Take-away: Sign-based MI-DP gets closer to achieving all three qualities of the privacy, utility, performance trilemma: FERRET trains up to 5x faster, achieves 3x lower perplexity compared to DPSGD and 1.2x greater than Non-DP, all while providing formal, mathematically provable privacy guarantees using zero additive noise. The results also show that, in certain instances, masked 1-bit updates can match non-private training utility while safeguarding data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。