用合成数据破解数据推断难题,让版权方能验证模型是否盗用数据。
Unlocking Post-hoc Dataset Inference with Synthetic Data
- 通过后缀补全任务训练生成器,合成高质量、多样化的替代数据集。
- 采用事后校准解决真实与合成数据的似然差距,检测准确率超90%。
- 适合关注数据版权保护的研究者和法律维权人员使用。
大型语言模型的强大能力主要源于其海量训练数据,这些数据常未经许可从网络抓取,侵犯了数据所有者的知识产权。数据推断(DI)提供了一种解决方案,可判断可疑数据集是否被用于训练,使数据所有者能够验证未经授权的使用。然而,现有方法需要一个已知未参与训练且分布相近的保留数据集,这类数据在实际中极少存在,严重限制了DI的应用。本文提出通过合成生成所需保留数据集来解决该问题。我们克服两大挑战:(1) 利用精心设计的后缀补全任务训练数据生成器,生成高保真、多样化的合成数据;(2) 通过事后校准弥合真实与合成数据间的似然差异。在多种文本数据集上的实验表明,使用生成数据作为保留集,可高置信度检测原始训练集,同时保持低误报率。该方法使版权方能够合法主张数据使用权,并在现实诉讼中展现出可靠性。代码已开源:https://github.com/sprintml/PostHocDatasetInference。
原文摘要 · Abstract (English)
The remarkable capabilities of Large Language Models (LLMs) can be mainly attributed to their massive training datasets, which are often scraped from the internet without respecting data owners' intellectual property rights. Dataset Inference (DI) offers a potential remedy by identifying whether a suspect dataset was used in training, thereby enabling data owners to verify unauthorized use. However, existing DI methods require a private set-known to be absent from training-that closely matches the compromised dataset's distribution. Such in-distribution, held-out data is rarely available in practice, severely limiting the applicability of DI. In this work, we address this challenge by synthetically generating the required held-out set. Our approach tackles two key obstacles: (1) creating high-quality, diverse synthetic data that accurately reflects the original distribution, which we achieve via a data generator trained on a carefully designed suffix-based completion task, and (2) bridging likelihood gaps between real and synthetic data, which is realized through post-hoc calibration. Extensive experiments on diverse text datasets show that using our generated data as a held-out set enables DI to detect the original training sets with high confidence, while maintaining a low false positive rate. This result empowers copyright owners to make legitimate claims on data usage and demonstrates our method's reliability for real-world litigations. Our code is available at https://github.com/sprintml/PostHocDatasetInference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。