提出一种无需依赖参数选择的隐私保护微调方法，解决低秩适配中噪声放大的核心问题。
PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
- 基于几何不变性设计隐私机制，避免低秩参数化带来的噪声放大
- 实现可证明的$(ε,δ)$-差分隐私，且噪声幅度可控
- 适合需要高隐私保障的模型微调场景，如医疗、金融数据
通过差分隐私（DP）的DP-SGD应用于低秩适配（LoRA）是隐私保护微调的自然方法。然而，LoRA的低秩参数化带来根本挑战：每个可训练更新表示为低秩矩阵 $Z = AB^ op$，但该分解本质上不可识别——多个因子对 $(A,B)$ 表示同一更新 $Z$。直接对因子应用DP-SGD会引入与规范相关的扰动，导致 $Z$ 上的噪声放大无界。本文提出PRISM，一种构造上保持规范不变的内在差分隐私机制，避免双线性噪声放大，并支持高效的低维噪声采样。此外，PRISM给出 $Z$ 上有效内在噪声的闭式表征，通过有界的规范不变扰动实现稳定的隐私-效用权衡。我们建立了PRISM的标准 $(ε,δ)$-差分隐私保证，并引入一种感知隐私、规范不变的自适应更新规则，防止自适应优化放大注入的隐私噪声，显著提升实际数值稳定性。
原文摘要 · Abstract (English)
Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix $Z = AB^\top$, but this factorization is inherently non-identifiable: many factor pairs $(A,B)$ represent the same update $Z$. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on $Z$, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on $Z$, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard $(ε,δ)$-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。