提出新型内存高效隐私训练方法,显著提升低内存下的模型性能。
Beyond Square Roots: Explicit Memory-Efficient Factorization for Multi-Epoch Private Learning

- 设计统一的γ-BIFR因子分解,兼顾内存效率与噪声相关性
- 在低带宽下比现有方法减少30%以上均方误差
- 适合资源受限场景下的多轮隐私训练,理论保障更严格
相关噪声机制是提升差分隐私模型训练效用的前沿方法,但严格的理论保证需依赖可分析的显式因子分解,而实际部署则要求内存高效。近期工作发展了带状逆因子分解,通过利用相关矩阵的带状结构同时满足两项需求。带宽决定了跨迭代噪声缓冲区的大小,从而控制效用与内存成本之间的权衡。现有方法凸显此权衡:DP-λCGD仅使用单步噪声缓冲区,内存效率高但效用提升有限;带状逆平方根(BISR)利用更大相关窗口,在大带宽下渐近最优,但在低带宽下表现不佳。本文提出γ-BIFR,作为两者的统一泛化。在低内存、低带宽场景下,γ-BIFR显著降低均方误差(RMSE)、放大均方误差及私有训练误差,并为多参与情况下的多轮训练提供更紧的理论保证。
原文摘要 · Abstract (English)
Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, analyzable factorizations, and practical deployment requires memory efficiency. Recent works have developed banded inverse factorizations, which address both requirements by exploiting a banded structure in the correlation matrix. The bandwidth controls the size of the noise buffer used to correlate noise across iterations, and thus governs the tradeoff between utility and memory cost. Existing factorizations highlight this tradeoff: DP-$λ$CGD achieves high memory efficiency by using only a one-step noise buffer, but this limits its utility gains, while the banded inverse square root (BISR) factorization exploits larger correlation windows and is asymptotically optimal for large bandwidths but performs poorly at low bandwidths. We propose $γ$-BIFR, a unified generalization of both factorizations. In the low-memory, low-bandwidth regime, $γ$-BIFR significantly improves RMSE, amplified RMSE, and private training performance, while yielding tighter theoretical guarantees for multi-participation error in multi-epoch training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。