通过分析模型预测与真实词的差距，提升预训练数据检测精度。
Gap-K%: Measuring Top-1 Prediction Gap for Detecting Pretraining Data
- 利用模型最可能预测词与真实词的对数概率差作为检测信号。
- 在WikiMIA和MIMIR上均超越现有方法，模型越小效果越显著。
- 适合关注LLM隐私与版权问题的研究者或安全工程师。
大型语言模型（LLMs）预训练语料库的不透明性引发严重的隐私与版权担忧，使得预训练数据检测成为关键挑战。现有先进方法多依赖词元似然，却常忽略目标词与模型顶1预测之间的差距，以及相邻词元间的局部相关性。本文提出Gap-K%，一种基于LLM预训练优化动态的新型检测方法。通过分析下一个词元预测目标，我们发现模型顶1预测与真实目标之间的差异会激发强梯度信号，这些信号在训练中被显式惩罚。受此启发，Gap-K%采用顶1预测词与目标词之间的对数概率差，并结合滑动窗口策略捕捉局部相关性，缓解词级波动。在WikiMIA与MIMIR基准上的大量实验表明，Gap-K%表现优异，跨不同模型规模与输入长度持续优于先前基线。
原文摘要 · Abstract (English)
The opacity of massive pretraining corpora in Large Language Models (LLMs) raises significant privacy and copyright concerns, making pretraining data detection a critical challenge. Existing state-of-the-art methods typically rely on token likelihoods, yet they often overlook the gap between the target token and the model's top-1 prediction, as well as local correlations between adjacent tokens. In this work, we propose Gap-K%, a novel pretraining data detection method grounded in the optimization dynamics of LLM pretraining. By analyzing the next-token prediction objective, we observe that discrepancies between the model's top-1 prediction and the target token induce strong gradient signals, which are explicitly penalized during training. Motivated by this, Gap-K% leverages the log probability gap between the top-1 predicted token and the target token, incorporating a sliding window strategy to capture local correlations and mitigate token-level fluctuations. Extensive experiments on the WikiMIA and MIMIR benchmarks demonstrate that Gap-K% achieves state-of-the-art performance, consistently outperforming prior baselines across various model sizes and input lengths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。