提出位置衰减重加权框架,提升大模型训练数据检测精度
PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection
- 基于自回归生成中早期高熵令牌记忆信号更强的发现,动态调整各位置得分权重
- 在多个基准上使主流方法检测准确率平均提升5.2%,最高达8.7%
- 无需训练、可即插即用,适合隐私审计与版权合规场景
大语言模型训练数据检测对保障数据隐私和版权合规至关重要,但在黑箱、零样本设置下仍具挑战性,尤其受限于计算资源与训练数据。现有基于似然的方法通常对词元级得分采用均匀加权,忽略了自回归生成中的信息论动态特性。本文提出并实证验证:记忆信号主要集中于高熵初始词元,随上下文累积而衰减。为此,我们引入无需训练、可即插即用的位置衰减重加权(PDR)框架,显式重加权词元级得分,强化早期信号,抑制后期噪声。大量实验表明,PDR可作为强先验,显著提升多种先进方法在多个基准上的性能。
原文摘要 · Abstract (English)
Detecting pre-training data in Large Language Models (LLMs) is crucial for auditing data privacy and copyright compliance, yet it remains challenging in black-box, zero-shot settings where computational resources and training data are scarce. While existing likelihood-based methods have shown promise, they typically aggregate token-level scores using uniform weights, thereby neglecting the inherent information-theoretic dynamics of autoregressive generation. In this paper, we hypothesize and empirically validate that memorization signals are heavily skewed towards the high-entropy initial tokens, where model uncertainty is highest, and decay as context accumulates. To leverage this linguistic property, we introduce Positional Decay Reweighting (PDR), a training-free and plug-and-play framework. PDR explicitly reweights token-level scores to amplify distinct signals from early positions while suppressing noise from later ones. Extensive experiments show that PDR acts as a robust prior and can usually enhance a wide range of advanced methods across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。