通过分析神经元激活模式,精准检测大模型记忆训练数据问题。
Detecting Memorization in Large Language Models
- 用神经元激活模式区分记忆与非记忆词汇,识别精准。
- 分类探针达到近似完美准确率,可抑制记忆不降性能。
- 适合研究模型隐私、可解释性及训练优化的开发者使用。
大型语言模型在自然语言处理中表现优异,但容易记忆训练数据片段,影响评估指标、引发隐私担忧并限制泛化能力。传统检测方法依赖输出概率或损失函数,常受常见语言模式干扰而精度不足。本文提出一种分析方法,通过考察模型内部神经元激活模式来精确识别记忆行为。通过识别区分记忆与非记忆标记的特定激活模式,训练分类探针实现近乎完美的准确率。该方法还可用于检测重复等其他机制,展现其通用性。干预这些激活可有效抑制记忆,且不损害整体性能,提升评估可信度,确保指标反映真实泛化能力。此外,该方法支持大规模标记词元与序列,对下一代AI模型的训练效率和效果有重要意义。研究成果增强了模型可解释性,提供了分析与控制大模型内部机制的实用工具。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved impressive results in natural language processing but are prone to memorizing portions of their training data, which can compromise evaluation metrics, raise privacy concerns, and limit generalization. Traditional methods for detecting memorization rely on output probabilities or loss functions, often lacking precision due to confounding factors like common language patterns. In this paper, we introduce an analytical method that precisely detects memorization by examining neuron activations within the LLM. By identifying specific activation patterns that differentiate between memorized and not memorized tokens, we train classification probes that achieve near-perfect accuracy. The approach can also be applied to other mechanisms, such as repetition, as demonstrated in this study, highlighting its versatility. Intervening on these activations allows us to suppress memorization without degrading overall performance, enhancing evaluation integrity by ensuring metrics reflect genuine generalization. Additionally, our method supports large-scale labeling of tokens and sequences, crucial for next-generation AI models, improving training efficiency and results. Our findings contribute to model interpretability and offer practical tools for analyzing and controlling internal mechanisms in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。