动态量化调度提升私密训练效率,精度损失低于2%。
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
- 按层动态选择量化范围,每轮轮换量化层
- 结合损失敏感度评估,减少量化误差影响
- 适用于隐私保护训练,尤其适合低精度硬件
差分隐私随机梯度下降(DP-SGD)及其自适应变体DP-Adam是保护敏感数据训练中用户隐私的强大技术。在训练过程中,将模型权重和激活值转换为低精度格式(即量化)可显著降低训练时间、能耗与成本,因此被广泛应用。本文首次揭示,与普通SGD相比,量化在差分隐私训练中导致更严重的精度下降,原因是噪声注入放大了量化方差,引发显著性能损失。为此,我们提出DPQuant,一种动态量化框架,通过自适应地在每轮选择不同子集的层进行量化来缓解该问题。其核心思想包括:(i) 每轮轮换量化层的随机采样策略,减少偏差;(ii) 基于差分隐私损失敏感度估计器的层优先级排序,识别可安全量化的层。该估计器仅消耗极小部分隐私预算,不破坏差分隐私保证。在ResNet18、ResNet50和DenseNet121上对多个数据集的实验表明,DPQuant始终优于静态量化基线,在保持验证精度下降小于2%的前提下,实现高达2.21倍的理论吞吐量提升,达到近似帕累托最优的精度-计算权衡。此外,该框架可扩展至DP-Adam并获得相似收益。
原文摘要 · Abstract (English)
Differentially-Private SGD (DP-SGD) and its adaptive variant DP-Adam are powerful techniques to protect user privacy when using sensitive data to train neural networks. During training, converting model weights and activations into low-precision formats, i.e., quantization, can drastically reduce training times, energy consumption, and cost, and is thus a widely used technique. In this work, we demonstrate for the first time that quantization causes significantly higher accuracy degradation in DP training compared to regular SGD. We observe that this is caused by noise injection, which amplifies quantization variance, leading to disproportionately large accuracy degradation. To address this challenge, we present DPQuant, a dynamic quantization framework that adaptively selects a changing subset of layers to quantize at each epoch. Our method combines two key ideas that effectively reduce quantization variance: (i) probabilistic sampling that rotates which layers are quantized every epoch, and (ii) loss-aware layer prioritization, which uses a differentially private loss sensitivity estimator to identify layers that can be quantized with minimal impact on model quality. This estimator consumes a negligible fraction of the overall privacy budget, preserving DP guarantees. Empirical evaluations on ResNet18, ResNet50, and DenseNet121 across a range of datasets demonstrate that DPQuant consistently outperforms static quantization baselines, achieving near Pareto-optimal accuracy-compute trade-offs and up to $2.21\times$ theoretical throughput improvements on low-precision hardware, with less than 2% drop in validation accuracy. We further show that our framework extends to DP-Adam with similar gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。