用CNN分析大模型记忆机制，发现多数记忆是猜的而非真回忆。
Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMs
- 用CNN分析大模型注意力权重，识别记忆类型
- 发现少样本记忆无独特注意力模式，多数是猜测
- 提出新分类法并可视化记忆区域位置
大型语言模型中的原样记忆是一个涉及多种机制的复杂现象。本文提出一种新方法，通过在大模型的注意力权重上训练卷积神经网络（CNN），评估现有记忆分类体系与解码过程中注意力权重的对齐程度。结果表明，现有分类体系表现不佳，无法反映注意力块内的不同机制。我们提出了一个新分类体系，最大化与注意力权重的对齐性，包含三类：利用语言建模能力猜测的记忆样本、因训练集中高重复性而被回忆的记忆样本，以及非记忆样本。研究发现，少样本原样记忆并不对应独特的注意力机制；且大量可提取样本实为模型猜测所得，应单独研究。最后，我们开发了一种定制化的视觉可解释性技术，定位每种记忆形式所涉及的注意力权重区域。
原文摘要 · Abstract (English)
Verbatim memorization in Large Language Models (LLMs) is a multifaceted phenomenon involving distinct underlying mechanisms. We introduce a novel method to analyze the different forms of memorization described by the existing taxonomy. Specifically, we train Convolutional Neural Networks (CNNs) on the attention weights of the LLM and evaluate the alignment between this taxonomy and the attention weights involved in decoding. We find that the existing taxonomy performs poorly and fails to reflect distinct mechanisms within the attention blocks. We propose a new taxonomy that maximizes alignment with the attention weights, consisting of three categories: memorized samples that are guessed using language modeling abilities, memorized samples that are recalled due to high duplication in the training set, and non-memorized samples. Our results reveal that few-shot verbatim memorization does not correspond to a distinct attention mechanism. We also show that a significant proportion of extractable samples are in fact guessed by the model and should therefore be studied separately. Finally, we develop a custom visual interpretability technique to localize the regions of the attention weights involved in each form of memorization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。