发现大模型推理能力主要来自输出投影层，而非整体模型。
Who Reasons in the Large Language Models?
- 通过诊断工具SfN分析模型内部行为，定位推理核心模块。
- 输出投影层（oproj）对数学推理起决定作用，其他模块主负责对话流畅性。
- 为模型可解释性提供新视角，适合关注模型优化与轻量化研究者。
尽管大语言模型表现优异，但赋予其新能力（如数学推理）的过程仍以经验为主且难以解释。一个关键问题是：推理能力源于整个模型、特定模块，还是过拟合的产物？本文假设，在训练良好的大语言模型中，推理能力主要来自Transformer多头自注意力机制中的输出投影模块（oproj）。为验证该假设，我们提出网络听诊器（SfN），一套用于探测和分析大语言模型内部行为的诊断工具。利用SfN，我们提供了间接和实证证据，表明oproj在实现推理中起核心作用，而其他模块则更多贡献于对话流畅性。这些发现为大语言模型可解释性提供了新视角，并开启了更精准训练策略的大门，有望实现更高效、更专业的语言模型。
原文摘要 · Abstract (English)
Despite the impressive performance of large language models (LLMs), the process of endowing them with new capabilities--such as mathematical reasoning--remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well-trained LLMs are primarily attributed to the output projection module (oproj) in the Transformer's multi-head self-attention (MHSA) mechanism. To support this hypothesis, we introduce Stethoscope for Networks (SfN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SfN, we provide both circumstantial and empirical evidence suggesting that oproj plays a central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer a new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。