利用大模型对已见文件的过度自信,检测其训练数据是否包含版权内容。
As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Files
- 通过捕捉不确定性模式区分训练过和未训练过的文件。
- 在LLaMA 7b上达到90.1%平衡准确率,优于现有方法超90%。
- 无需人工调参,适用于多种模型架构,适合版权审计场景。
大型语言模型(LLMs)的强大语言能力源于对海量数据的训练,其中常包含受版权保护的内容,引发未经授权使用的担忧。尽管成员推理攻击(MIAs)可为检测此类违规提供可能,但现有方法受限于大模型的过度自信、缺乏真实训练数据以及依赖经验设定阈值等问题。本文提出COPYCHECK,一种利用不确定性信号检测训练集中是否存在版权内容的新框架。该方法将大模型的过度自信转化为优势,通过捕捉可靠的不确定性模式,有效区分“已见”(训练数据)与“未见”(非训练数据)内容。COPYCHECK采用双重策略:(1)将文件分段为小片段以降低对大规模训练数据的依赖;(2)基于不确定性的无监督聚类,消除对经验阈值的需求。实验表明,该方法在LLaMA 7b上实现90.1%的平均平衡准确率,在LLaMA2 7b上达91.6%,相比最优基线提升超90%,最高达93.8%。其性能在GPT-J 6B上也保持优异,展现了强泛化能力。本工作首次将不确定性应用于大模型版权检测,为训练数据透明性提供了实用工具。
原文摘要 · Abstract (English)
The remarkable language ability of Large Language Models (LLMs) stems from extensive training on vast datasets, often including copyrighted material, which raises serious concerns about unauthorized use. While Membership Inference Attacks (MIAs) offer potential solutions for detecting such violations, existing approaches face critical limitations and challenges due to LLMs' inherent overconfidence, limited access to ground truth training data, and reliance on empirically determined thresholds. We present COPYCHECK, a novel framework that leverages uncertainty signals to detect whether copyrighted content was used in LLM training sets. Our method turns LLM overconfidence from a limitation into an asset by capturing uncertainty patterns that reliably distinguish between ``seen" (training data) and ``unseen" (non-training data) content. COPYCHECK further implements a two-fold strategy: (1) strategic segmentation of files into smaller snippets to reduce dependence on large-scale training data, and (2) uncertainty-guided unsupervised clustering to eliminate the need for empirically tuned thresholds. Experiment results show that COPYCHECK achieves an average balanced accuracy of 90.1% on LLaMA 7b and 91.6% on LLaMA2 7b in detecting seen files. Compared to the SOTA baseline, COPYCHECK achieves over 90% relative improvement, reaching up to 93.8\% balanced accuracy. It further exhibits strong generalizability across architectures, maintaining high performance on GPT-J 6B. This work presents the first application of uncertainty for copyright detection in LLMs, offering practical tools for training data transparency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。