不同Transformer模型记事实的方式差异显著，揭示了架构设计的关键影响。
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms
- 对比GPT、LLaMA等多模型，分析事实记忆的编码位置与机制。
- 发现Qwen模型中早期注意力层比MLP层更关键，突破既有认知。
- 对可解释性与模型编辑有启发，适合关注模型内部机理的研究者。
理解基于Transformer的语言模型如何存储和检索事实关联，对于提升可解释性及实现针对性模型编辑至关重要。以往研究主要针对GPT类模型，发现早期层中的MLP模块是事实回忆的关键贡献者。然而，这些发现是否在不同自回归架构中具有普适性仍不明确。为此，我们对包括GPT、LLaMA、Qwen和DeepSeek在内的多个模型进行了全面的事实回忆评估，分析事实信息在何处被编码及如何被访问。结果表明，Qwen系列模型的行为与以往模式不同：最早层的注意力模块对事实回忆的贡献大于MLP模块。研究提示，在自回归Transformer家族内部，架构差异可能导致根本不同的事实回忆机制。
原文摘要 · Abstract (English)
Understanding how Transformer-based language models store and retrieve factual associations is critical for improving interpretability and enabling targeted model editing. Prior work, primarily on GPT-style models, has identified MLP modules in early layers as key contributors to factual recall. However, it remains unclear whether these findings generalize across different autoregressive architectures. To address this, we conduct a comprehensive evaluation of factual recall across several models -- including GPT, LLaMA, Qwen, and DeepSeek -- analyzing where and how factual information is encoded and accessed. Consequently, we find that Qwen-based models behave differently from previous patterns: attention modules in the earliest layers contribute more to factual recall than MLP modules. Our findings suggest that even within the autoregressive Transformer family, architectural variations can lead to fundamentally different mechanisms of factual recall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。