解析Transformer如何完成逻辑推理，发现关键神经机制。
From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits
- 通过分析三段论提示，识别出支持逻辑推理的多个神经电路。
- 五注意力头组成的电路可实现原模型90%以上的性能。
- 揭示了模型生成输入中不存在的否定词的内在机制。
基于Transformer的语言模型能执行多种任务，机械可解释性（MI）旨在逆向工程完成任务的组件以理解其行为。以往的MI研究聚焦于间接宾语识别（IOI）等语言任务。本文研究GPT-2 small在处理二值真值判断时的表现，通过分析三段论类提示（如“陈述A为真。陈述B与陈述A一致。陈述B是”），该任务比IOI需更复杂的逻辑推理。通过对不同难度的三段论任务分析，我们识别出多个可机械解释模型逻辑推理能力的电路，并发现支持任务完成的二值机制，包括通过负向注意力头生成输入中未出现的否定词。使用忠实度指标评估表明，由五个注意力头组成的电路实现了超过90%的原始模型性能。结合已有IOI分析，我们对特定注意力头和MLP在语言模型中的作用提供了新见解，推动了对模型推理的理解，并支持未来机械可解释性研究。
原文摘要 · Abstract (English)
Transformer-based language models (LMs) can perform a wide range of tasks, and mechanistic interpretability (MI) aims to reverse engineer the components responsible for task completion to understand their behavior. Previous MI research has focused on linguistic tasks such as Indirect Object Identification (IOI). In this paper, we investigate the ability of GPT-2 small to handle binary truth values by analyzing its behavior with syllogistic prompts, e.g., "Statement A is true. Statement B matches statement A. Statement B is", which requires more complex logical reasoning compared to IOI. Through our analysis of several syllogism tasks of varying difficulty, we identify multiple circuits that mechanistically explain GPT-2's logical-reasoning capabilities and uncover binary mechanisms that facilitate task completion, including the ability to produce a negated token not present in the input prompt through negative heads. Our evaluation using a faithfulness metric shows that a circuit comprising five attention heads achieves over 90% of the original model's performance. By relating our findings to IOI analysis, we provide new insights into the roles of specific attention heads and MLPs in LMs. These insights contribute to a broader understanding of model reasoning and support future research in mechanistic interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。