提出可规模化检测大模型数据记忆的工具MemHunter，解决隐私泄露风险。
MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs
- 训练诱导记忆的LLM，用假设检验实现全数据集级检测
- 在有限时间内比现有方法多提取40%训练数据，搜索时间减少80%
- 首次实现数据集级记忆检测，适合数据所有者验证模型隐私风险
大型语言模型会记忆并复现训练数据内容，引发重大隐私担忧，尤其在网页规模数据集下。现有检测方法多为样本级，依赖手动或优化生成的记忆诱导提示，逐样本执行计算成本过高，难以扩展到数据集层面。实际场景中，数据所有者需验证模型是否未经授权地记住了其数据。为此，本文提出MemHunter：通过训练一个记忆诱导型语言模型，并结合假设检验，在无需样本级诱导提示的情况下，高效实现数据集级别的记忆检测。在Pythia和Llama等模型上的实验表明，该方法在受限时间内比现有方法多提取40%的训练数据，集成为插件时搜索时间降低高达80%。关键突破在于首次实现数据集级记忆检测，为基于大规模数据集的LLM提供关键隐私风险评估工具。
原文摘要 · Abstract (English)
Large language models (LLMs) have been shown to memorize and reproduce content from their training data, raising significant privacy concerns, especially with web-scale datasets. Existing methods for detecting memorization are primarily sample-specific, relying on manually crafted or discretely optimized memory-inducing prompts generated on a per-sample basis, which become impractical for dataset-level detection due to the prohibitive computational cost of iterating through all samples. In real-world scenarios, data owners may need to verify whether a susceptible LLM has memorized their dataset, particularly if the LLM may have collected the data from the web without authorization. To address this, we introduce MemHunter, which trains a memory-inducing LLM and employs hypothesis testing to efficiently detect memorization at the dataset level, without requiring sample-specific memory inducing. Experiments on models like Pythia and Llama demonstrate that MemHunter can extract up to 40% more training data than existing methods under constrained time resources and reduce search time by up to 80% when integrated as a plug-in. Crucially, MemHunter is the first method capable of dataset-level memorization detection, providing a critical tool for assessing privacy risks in LLMs powered by large-scale datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。