arXiv:2605.24173cs.CLcs.AI2026-05

扩散语言模型训练数据可被更高效提取,因可任意位置补全文本。

Extracting Training Data from Diffusion Language Models via Infilling

  • 提出'填空式提取'新方法,支持任意位置掩码,突破传统前缀限制。
  • 边缘掩码提取效果是前缀的三倍,双向推理暴露更多数据泄露风险。
  • 即使删除敏感信息,仍能比自回归模型更准确还原隐私数据。

大型语言模型中的记忆化问题长期仅通过前缀条件提取研究,这适用于自回归模型。然而,扩散语言模型(DLMs)可在任意位置去噪掩码词元,因此仅依赖前缀探测只能反映其记忆化的一小部分,严重低估了训练数据被提取的风险。为更真实评估DLM中训练数据的可提取性,我们引入了 extit{infilling extraction}——一种由任意二值掩码参数化的数据提取协议,涵盖前缀探测并体现DLM的双向归纳偏置。在LLaDA-8B和Dream-7B上,针对五种提取模式、三种训练流程和三个语料库(覆盖完整与部分泄漏),发现掩码几何形状决定可提取性:边缘条件掩码提取完整序列的能力是前缀条件的三倍;双向访问打开了自回归模型无法触及的数据通道。特别地,我们证明,一个能访问经脱敏处理训练数据的现实攻击者,反而能在从DLM中提取被遮蔽的邮箱地址时获得比同规模自回归模型更高的召回率。解码过程的可调参数显著影响提取性能,而后续监督微调阶段无法消除先前的记忆化现象。

原文摘要 · Abstract (English)

Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce \emph{infilling extraction}, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks \emph{extract up to three times more} verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, we show that a realistic adversary with access to training data where personally identifiable information has been redacted, can even achieve higher recall on extracting redacted email addresses from DLMs than from scale-matched autoregressive models. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised finetuning stage does not eliminate the prior memorization.

扩散模型数据提取隐私安全记忆化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。