arXiv:2606.01849cs.LGcs.CL2026-06

测试差分隐私文本能否带来真实新能力,发现现有方法基本无效。

ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?

论文配图:ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?
图 1 · 摘自论文原文
  • 构建可自动更新的持续性评测基准,强制要求模型从原始数据学新知识。
  • 非私有合成数据能有效迁移原数据知识,而主流差分隐私方法在ε=100时仍失败。
  • 适合关注隐私生成与模型能力边界的研究者,尤其对可信数据合成感兴趣者。

差分隐私(DP)文本合成有望解锁敏感语料库用于模型训练,但尚不清楚其能否传递原语料库中独有的真实新知识与能力。现有评估依赖几乎无需训练即可解决的任务,无法证明DP合成数据可替代原始数据访问。为此,我们提出ContinuousBench:一个持续自动更新的评测基准,用于衡量从DP合成文本中获得的能力提升。每季度发布一次,配对从未见过的训练语料与对应的问答集,该问答集满足:(1)无原始语料则无法解答;(2)在差分隐私条件下可学习,因目标知识由数百条独立记录支持。研究人员使用原始语料生成DP合成数据,并在统一训练与评估流程下测量性能提升。我们设立两个赛道:Geminon(虚构生物的程序生成数据集)和News(新爬取的公共新闻流)。尽管传统基准已接近饱和,但在ContinuousBench上,非私有合成数据能显著迁移原语料知识,而最先进的DP合成方法即使在ε=100时也普遍表现不佳。

原文摘要 · Abstract (English)

Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora. This is because existing evaluations rely on tasks that are nearly solvable without training, so strong benchmark performance does not establish that DP synthesis can substitute original data access. Thus, we introduce ContinuousBench, a continuously and automatically-regenerated benchmark that measures capability gain from DP synthetic text. Each quarter, a new release pairs a never-before-seen training corpus with a derived QA set, constructed to be: (1) unsolvable sans-corpus; and (2) learnable under DP, as the tested knowledge is supported by hundreds of independent records. Researchers produce DP synthetic data from the training corpus and run our standardized training and evaluation harness on their synthetic data to measure gains. We instantiate two tracks: Geminon, a procedurally-generated dataset about fictional creatures; and News, a stream of newly crawled public news articles. Although standard benchmarks are nearly saturated, on ContinuousBench we find that non-private synthesis transfers substantial knowledge from the original corpus, while state-of-the-art DP synthesis methods generally fail to do so, even at $\varepsilon=100$.

差分隐私文本生成评测基准知识迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。