揭示文本差分隐私生成中的表示空间扭曲问题
Inspecting the Representation Manifold of Differentially-Private Text
- 通过估计不同隐私预算下的内在维度，分析文本重写对表示流形的影响
- 词级方法严重增加流形复杂度，句级方法更贴近人类写作的拓扑结构
- 掩码重写优于自回归重写，因其减少错误词汇传播导致的结构膨胀
文本差分隐私近年来多采用语言模型进行文本重写与温度采样，以平衡隐私与效用。然而，此类方法在表示空间中的几何失真——包括结构与复杂性变化——仍未被充分探索。本文通过估算不同隐私预算下重写文本的内在维度发现：词级方法显著提升表示流形复杂度；而句级方法生成的重写文本其流形拓扑更接近人类撰写的同义改写。在句级方法中，相比自回归重写，掩码重写更能有效保留结构复杂性，表明自回归生成会因不自然的词选择引入误差，并逐层累积放大，导致表示空间膨胀。
原文摘要 · Abstract (English)
Differential Privacy (DP) for text has recently taken the form of text paraphrasing using language models and temperature sampling to better balance privacy and utility. However, the geometric distortion of DP regarding the structure and complexity in the representation space remains unexplored. By estimating the intrinsic dimension of paraphrased text across varying privacy budgets, we find that word-level methods severely raise the representation manifold, while sentence-level methods produce paraphrases whose manifolds are topologically more consistent with human-written paraphrases. Among sentence-level methods, masked paraphrasing, compared to causal paraphrasing, demonstrates superior preservation of structural complexity, suggesting that autoregressive generation propagates distortions from unnatural word choices that cascade and inflate the representation space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。