首次将差分隐私用于语音识别预训练，兼顾隐私与模型性能。
Training Large ASR Encoders with Differential Privacy
- 用梯度层冻结技术优化隐私-性能-计算权衡
- 在(10, 1e-9) DP下实现3.78/8.41的词错率
- 适合关注语音模型隐私保护的研究者
自监督学习（SSL）方法在大规模语音模型中已被证明对自动语音识别（ASR）非常有效。随着大模型公开部署需求上升，训练数据中敏感信息的意外记忆和泄露引发关注。本文首次将差分隐私（DP）应用于基于SOTA Conformer的语音编码器预训练，并研究其在下游ASR任务中的表现，假设微调数据是公开的。我们针对BEST-RQ预训练方法探究了其对DP噪声的容忍度，并提出一种新型模型剪枝方法——基于梯度的层冻结，显著提升了隐私-效用-计算的平衡。在低数据量场景下，模型在(10, 1e^-9)-DP下取得3.78/8.41的LibriSpeech test-clean/other词错率；在高数据量场景下，于(10, 7.9e^-11)-DP下达到2.81/5.89的词错率。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) methods for large speech models have proven to be highly effective at ASR. With the interest in public deployment of large pre-trained models, there is a rising concern for unintended memorization and leakage of sensitive data points from the training data. In this paper, we apply differentially private (DP) pre-training to a SOTA Conformer-based encoder, and study its performance on a downstream ASR task assuming the fine-tuning data is public. This paper is the first to apply DP to SSL for ASR, investigating the DP noise tolerance of the BEST-RQ pre-training method. Notably, we introduce a novel variant of model pruning called gradient-based layer freezing that provides strong improvements in privacy-utility-compute trade-offs. Our approach yields a LibriSpeech test-clean/other WER (%) of 3.78/ 8.41 with ($10$, 1e^-9)-DP for extrapolation towards low dataset scales, and 2.81/ 5.89 with (10, 7.9e^-11)-DP for extrapolation towards high scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。