用带密钥水印的改写内容，检测文本是否被纳入大模型训练数据。
STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings
- 对原文生成多版带唯一密钥水印的改写文本
- 仅公开一版，其余私藏后比对模型概率差异
- 可精准识别占比不足0.001%的隐藏数据
由于大量公开文本被用于预训练大语言模型（LLMs），内容创作者日益担忧其专有数据在未获署名或授权的情况下被用于模型训练。这一问题同样影响基准测试集的完整性。本文提出STAMP框架，用于检测数据集成员资格——即判断某数据集是否被包含在大型语言模型的预训练语料中。给定原始内容，该方法首先生成多个改写版本，每个版本嵌入唯一密钥的水印。其中一版公开发布，其余保留私密。创作者随后通过配对统计检验比较模型对公开与私密版本的似然值，以证明成员身份。实验表明，该框架能成功检测出在训练数据中仅出现一次且占总词元少于0.001%的四个基准数据集的污染情况，优于多个现有检测与数据推断基线。我们验证了STAMP在保持原文语义和实用性方面表现良好。将该方法应用于真实场景，确认论文摘要与博客文章确已被纳入预训练语料。
原文摘要 · Abstract (English)
Given how large parts of publicly available text are crawled to pretrain large language models (LLMs), data creators increasingly worry about the inclusion of their proprietary data for model training without attribution or licensing. Their concerns are also shared by benchmark curators whose test-sets might be compromised. In this paper, we present STAMP, a framework for detecting dataset membership-i.e., determining the inclusion of a dataset in the pretraining corpora of LLMs. Given an original piece of content, our proposal involves first generating multiple rephrases, each embedding a watermark with a unique secret key. One version is to be released publicly, while others are to be kept private. Subsequently, creators can compare model likelihoods between public and private versions using paired statistical tests to prove membership. We show that our framework can successfully detect contamination across four benchmarks which appear only once in the training data and constitute less than 0.001% of the total tokens, outperforming several contamination detection and dataset inference baselines. We verify that STAMP preserves both the semantic meaning and utility of the original data. We apply STAMP to two real-world scenarios to confirm the inclusion of paper abstracts and blog articles in the pretraining corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。