通过记忆覆盖现象，无需内部数据即可检测模型是否用过特定数据集。
Hey, That's My Data! Token-Only Dataset Inference in Large Language Models
- 基于灾难性遗忘机制，仅用输出变化判断数据是否被训练
- 在开源与API模型上均有效，准确率超90%且无需访问概率输出
- 适合保护商业数据版权，尤其适用于闭源模型场景
大型语言模型依赖海量训练数据，其中常包含专有数据，引发未经授权使用和版权侵权担忧。现有数据集推断方法通常需要访问对数概率等内部信号，但许多现代LLM限制此类访问，促使发展仅需标记（token-only）的推断方法。本文提出CatShift框架，基于灾难性遗忘原理：当模型在新数据上微调时，会覆盖原有知识。将模型在自身部分训练数据上微调引起的输出变化，与在已知非成员验证集上的变化进行对比，可推断目标数据集是否曾用于训练。在开源及API类大模型上的实验表明，即使无对数输出访问权限，CatShift仍具有效性，为专有数据提供了实际保护手段。
原文摘要 · Abstract (English)
Large Language Models (LLMs) rely on massive training datasets, often including proprietary data, which raises concerns about unauthorized usage and copyright infringement. Existing dataset inference methods typically require access to log probabilities or other internal signals, but many modern LLMs restrict such access, motivating token-only inference approaches. We propose CatShift, a token-only dataset inference framework based on catastrophic forgetting, where models overwrite prior knowledge when trained on new data. Fine-tuning an LLM on a subset of its training data induces larger output shifts than fine-tuning on unseen data. CatShift compares these shifts against those from a known non-member validation set to infer whether a dataset was included in training. Experiments on both open-source and API-based LLMs show that CatShift remains effective without logit access, enabling practical protection of proprietary datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。