用改写文本给数据打水印，让版权检测更可靠
Perturb Your Data: Paraphrase-Guided Training Data Watermarking
- 用大模型改写数据并选匹配原内容得分的版本，避免分布偏移
- 检测时比对模型概率与评分模型，差异超九个数量级
- 适合数据所有者在发布前部署，抗大规模训练干扰
训练数据检测对保护版权和数据许可至关重要，因为大型语言模型（LLM）通常基于从互联网抓取的海量文本语料进行训练。我们提出SPECTRA，一种水印方法，即使水印数据仅占训练语料的0.001%以下，也能实现可靠检测。SPECTRA通过使用LLM对文本进行改写，并基于独立评分模型判断每个改写版本的合理性，选择与原始文本得分相近的改写结果，以避免引入分布偏移。为检测可疑模型是否曾使用水印数据训练，我们将其标记概率与评分模型进行比较。实验表明，SPECTRA在区分训练数据与非训练数据时，实现了超过九个数量级的一致p值差距，优于所有对比基线。SPECTRA为数据所有者提供了可扩展、可在发布前部署的水印方案，即使经过大规模LLM训练仍有效。
原文摘要 · Abstract (English)
Training data detection is critical for enforcing copyright and data licensing, as Large Language Models (LLM) are trained on massive text corpora scraped from the internet. We present SPECTRA, a watermarking approach that makes training data reliably detectable even when it comprises less than 0.001% of the training corpus. SPECTRA works by paraphrasing text using an LLM and assigning a score based on how likely each paraphrase is, according to a separate scoring model. A paraphrase is chosen so that its score closely matches that of the original text, to avoid introducing any distribution shifts. To test whether a suspect model has been trained on the watermarked data, we compare its token probabilities against those of the scoring model. We demonstrate that SPECTRA achieves a consistent p-value gap of over nine orders of magnitude when detecting data used for training versus data not used for training, which is greater than all baselines tested. SPECTRA equips data owners with a scalable, deploy-before-release watermark that survives even large-scale LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。