隐藏状态隐私与效用难以兼顾,中间地带为空白。
Hidden-State Privacy Has an Empty Middle

- 提出对角逆Fisher机制,实现最优隐私与效用权衡。
- 实测1536种协方差中无一同时满足中等效用与中等隐私。
- 适合关注模型隐私泄露风险的研究者与系统设计者。
在测试的1536个高斯发布协方差中,无一能在对抗性检索攻击下同时实现中等效用与中等隐私。我们证明了互补的Fisher球下界:每个具有O(1) Fisher效用的满秩高斯发布,总存在一个方向其马哈拉诺比斯信号随隐藏层宽度线性增长,从而排除了该类中均匀高斯安全性的可能,并与实验结果一致。对角逆Fisher发布Σ⋆_diag(K) = (2K/d) diag(1/F_ii) 是在第一阶KL预算K下的唯一极小极大最优对角机制,且在32层模型网格上所有点的最坏攻击者前1准确率≤0.001,但位于隐私/效用边界而非中间区域。广义特征值机制在欧氏检索下实现13倍帕累托改进,但在自适应马哈拉诺比斯攻击下退化至100%前1准确率;全轨迹序列逆解器可恢复94%原始GPT-2前缀,但在Σ_diag下为0%。从零训练的分存变压器在90M参数量时达到G_Mah ∈ [20, 33],在固定令牌语言建模损失惩罚下,从30M到1B参数量均保持6–24倍优势,预训练模型最高仅达9.3。这些结果将隐藏状态发布问题从高斯类内的机制设计,重新定义为架构与发布联合设计。
原文摘要 · Abstract (English)
Of $1{,}536$ Gaussian release covariances we tested for single-layer hidden-state privacy, zero achieve both moderate utility and moderate privacy against an adaptive retrieval attacker. We prove a complementary Fisher-ball lower bound: every full-rank Gaussian release at $O(1)$ Fisher utility admits a direction whose Mahalanobis signal grows linearly in hidden width, ruling out uniform Gaussian safety in the class and matching the empirical empty middle. The diagonal inverse-Fisher release $Σ^\star_{\mathrm{diag}}(\mathcal{K}) = (2\mathcal{K}/d)\,\mathrm{diag}(1/F_{ii})$ is the unique minimax-optimal diagonal mechanism at first-order KL budget $\mathcal{K}$ and the only release with worst-attacker top-1 $\le 0.001$ at every point of a 32 model-layer grid, but it sits on a privacy/utility edge rather than filling the middle. A generalized-eigen mechanism reaching $13\times$ Pareto reduction under Euclidean retrieval collapses to $100\%$ top-1 under the adaptive Mahalanobis attacker, and a full-trajectory sequence inverter recovers $94\%$ of clean GPT-2 prefixes but $0\%$ under $Σ_{\mathrm{diag}}$. A split-memory transformer trained from scratch reaches $G_{\mathrm{Mah}} \in [20, 33]$ at 90M and maintains a $6$--$24\times$ advantage over same-budget GPT baselines from 30M to 1B at a fixed-token language-modeling loss penalty; pretrained models top out at 9.3. These results reframe hidden-state release from mechanism-design within the Gaussian class to architecture or release co-design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。