通过参数裁剪提升语言模型的隐私-效用平衡
Nonparametric Variational Differential Privacy via Embedding Parameter Clipping
- 基于瑞尼散度上界推导出后验参数裁剪规则
- 裁剪后模型隐私界更紧，下游任务性能更高
- 适合需要强隐私保障的生成式AI应用
非参数变分信息瓶颈（NVIB）是构建隐私保护语言模型的基础。然而，学习到的潜在表示可能漂移到高信息量区域，导致隐私保障差且训练中出现数值不稳定性。本文提出一种从最小化瑞尼散度（RD）上界推导出的参数裁剪策略，对后验均值、方差和混合权重施加理论约束。在基于NVIB的模型上应用该方法，并与无约束基线对比。实验表明，裁剪模型始终获得更紧的RD上界（即更强隐私），同时在多个下游任务中表现更优。本工作提供了一种简单但有效的改进变分模型隐私-效用权衡的方法，增强了模型鲁棒性与实用性。
原文摘要 · Abstract (English)
The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。