揭示Transformer+思维链如何隐式实现有限状态自动机
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking
- 通过分析模型内部组件，发现晚期MLP神经元构成状态追踪电路
- 在多种挑战场景下，状态追踪准确率接近100%
- 适合研究大模型内部机制与可解释性的人士阅读
思维链（CoT）显著提升大型语言模型在各类任务中的表现，已有研究证明其理论上可增强模型表达能力。然而，对Transformer+CoT所能学习到的算法机制仍缺乏深入理解。本文主要贡献包括：（1）评估Transformer+CoT及其变体的状态追踪能力，验证了CoT的有效性；（2）识别出负责追踪世界状态的神经电路，发现晚期层的MLP神经元起关键作用。提出压缩度与区分度两个指标，结果显示各状态对应的神经元集合实现近100%的追踪准确率，表明模型中隐含了有限状态自动机（FSA）结构；（3）进一步探索三种挑战场景：跳过中间步骤、引入数据噪声以及长度泛化测试。结果表明，Transformer+CoT能学习出鲁棒的算法（即FSA），在复杂条件下仍具稳定性。代码已公开于https://github.com/IvanChangPKU/FSA。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) significantly enhances the performance of large language models (LLMs) across a wide range of tasks, and prior research shows that CoT can theoretically increase expressiveness. However, there is limited mechanistic understanding of the algorithms that Transformer+CoT can learn. Our key contributions are: (1) We evaluate the state tracking capabilities of Transformer+CoT and its variants, confirming the effectiveness of CoT. (2) Next, we identify the circuit (a subset of model components, responsible for tracking the world state), indicating that late-layer MLP neurons play a key role. We propose two metrics, compression and distinction, and show that the neuron sets for each state achieve nearly 100% accuracy, providing evidence of an implicit finite state automaton (FSA) embedded within the model. (3) Additionally, we explore three challenging settings: skipping intermediate steps, introducing data noises, and testing length generalization. Our results demonstrate that Transformer+CoT learns robust algorithms (FSAs), highlighting its resilience in challenging scenarios. Our code is available at https://github.com/IvanChangPKU/FSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。