通过词元概率排序相关性，验证大模型未使用特定数据集。
Detecting Non-Membership in LLM Training Data via Rank Correlations
- 利用模型输出的词元概率排序相关性判断数据是否未被训练
- 在所有测试数据集上均可靠排除成员身份，且无误报
- 适用于版权合规与信任验证，适合模型审计人员使用
随着大语言模型（LLMs）训练数据规模持续扩大且愈发不透明，确定哪些数据被用于训练已成为版权保护、合规审计和用户信任的关键。现有研究多关注检测数据是否被纳入训练（成员身份推断），但对验证某数据集未被使用这一互补问题关注较少。本文提出PRISM，一种仅需灰盒访问模型输出对数概率即可检测数据集级非成员身份的测试方法。核心洞察是：两个均未见过某数据集的模型，在归一化词元对数概率上的排序相关性高于其中一个模型曾训练过该数据集的情况。基于此构建相关性测试，实证表明PRISM能可靠排除所有测试数据集的成员身份，且无误报，为验证特定数据集未被纳入大模型训练提供了有效框架。
原文摘要 · Abstract (English)
As large language models (LLMs) are trained on increasingly vast and opaque text corpora, determining which data contributed to training has become essential for copyright enforcement, compliance auditing, and user trust. While prior work focuses on detecting whether a dataset was used in training (membership inference), the complementary problem -- verifying that a dataset was not used -- has received little attention. We address this gap by introducing PRISM, a test that detects dataset-level non-membership using only grey-box access to model logits. Our key insight is that two models that have not seen a dataset exhibit higher rank correlation in their normalized token log probabilities than when one model has been trained on that data. Using this observation, we construct a correlation-based test that detects non-membership. Empirically, PRISM reliably rules out membership in training data across all datasets tested while avoiding false positives, thus offering a framework for verifying that specific datasets were excluded from LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。