通过电路发现，揭示大模型记忆文本的触发与维持机制。
Understanding Verbatim Memorization in LLMs Through Circuit Discovery
- 用对比数据集定位模型生成分歧点，识别记忆触发与维持电路。
- 记忆触发电路可启动并维持记忆，仅维持电路无法启动记忆。
- 防记忆机制跨领域有效，促记忆机制更依赖上下文。
大模型中记忆行为——即训练数据的原文重现——背后的机制仍不明确。究竟是网络的哪一部分决定了何时开始检索一段被认为是记忆起始的内容？模型在生成记忆内容与非记忆内容时的行为有何不同？本文从机械可解释性视角出发，利用Transformer电路（即执行特定功能的最小计算子图）回答这些问题。通过精心设计的对比数据集，我们识别出模型生成出现分歧的位置，并分离出负责记忆两个不同方面（触发与维持）的特定电路。研究发现，能够启动记忆的电路同样能维持记忆，而仅能维持记忆的电路无法触发记忆。有趣的是，防止记忆的机制在不同文本领域间具有强鲁棒性，而促进记忆的机制则更依赖具体上下文。
原文摘要 · Abstract (English)
Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token that we would consider as start of memorization sequence? How exactly is the models' behaviour different when producing memorized sentence vs non-memorized? In this work we approach these questions from mechanistic interpretability standpoint by utilizing transformer circuits -- the minimal computational subgraphs that perform specific functions within the model. Through carefully constructed contrastive datasets, we identify points where model generation diverges from memorized content and isolate the specific circuits responsible for two distinct aspects of memorization. We find that circuits that initiate memorization can also maintain it once started, while circuits that only maintain memorization cannot trigger its initiation. Intriguingly, memorization prevention mechanisms transfer robustly across different text domains, while memorization induction appears more context-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。