检测大模型训练数据泄露，发现现有方法难以可靠识别。
Detecting Data Contamination in Large Language Models
- 采用统一数据集对比多种黑盒成员推断方法
- 所有方法在主流大模型上AUC仅约0.5，效果接近随机
- 先进模型泛化能力强，更难被检测，适合安全与合规研究者
大型语言模型（LLMs）训练依赖海量数据，其中部分可能来自受版权保护的来源。成员推断攻击（MIA）旨在识别这些文档是否被纳入模型训练语料。黑盒MIA通常需大量数据操作，导致比较困难。本文在黑盒假设下研究当前最优（SOTA）MIA方法，并使用统一数据集进行对比，以评估其在主流大模型上可靠检测成员资格的能力。此外，提出一种新方法——熟悉度排序（Familiarity Ranking），展示黑盒MIA的一种可行路径，使大模型在表达中更具自由度，从而更好地理解其推理过程。结果表明，所有方法在多个大模型上的AUC-ROC均约为0.5，无法可靠检测成员身份。更先进模型表现出更高的真阳性率（TPR）和假阳性率（FPR），凸显其更强的推理与泛化能力，进一步证明黑盒MIA检测成员资格的难度。
原文摘要 · Abstract (English)
Large Language Models (LLMs) utilize large amounts of data for their training, some of which may come from copyrighted sources. Membership Inference Attacks (MIA) aim to detect those documents and whether they have been included in the training corpora of the LLMs. The black-box MIAs require a significant amount of data manipulation; therefore, their comparison is often challenging. We study state-of-the-art (SOTA) MIAs under the black-box assumptions and compare them to each other using a unified set of datasets to determine if any of them can reliably detect membership under SOTA LLMs. In addition, a new method, called the Familiarity Ranking, was developed to showcase a possible approach to black-box MIAs, thereby giving LLMs more freedom in their expression to understand their reasoning better. The results indicate that none of the methods are capable of reliably detecting membership in LLMs, as shown by an AUC-ROC of approximately 0.5 for all methods across several LLMs. The higher TPR and FPR for more advanced LLMs indicate higher reasoning and generalizing capabilities, showcasing the difficulty of detecting membership in LLMs using black-box MIAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。