研究发现,模型‘精通’后并未获得新推理能力,只是把记忆整合进已有路径。
Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers
- 通过机制分析发现,精通前后推理路径一致
- 高准确率与特定推理路径可独立出现,不互相绑定
- 即使成熟电路也难迁移新知识,说明未真正掌握组合逻辑
大型语言模型在事实检索上表现优异,但在复合任务中常遭遇“两跳推理困境”。近期研究指出,参数共享的Transformer在长期训练的‘精通’阶段会形成‘泛化电路’,从而弥合这一差距。本文开展机制性研究,评估该电路在知识吸收与迁移中的作用。结果表明:(i)非精通与精通模型对分布内复合查询的推理路径相同,说明‘泛化电路’并非突然获得新推理范式,而是将记忆化的原子事实融入已存在的推理路径;(ii)在特定数据条件下,高精度与特定推理路径的形成可独立发生,二者无必然关联;(iii)即便电路成熟,其在引入新知识时仍表现出有限迁移能力,表明‘精通’后的Transformer并未实现对组合逻辑的完全掌握。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) excel at factual retrieval, they often struggle with the "curse of two-hop reasoning" in compositional tasks. Recent research suggests that parameter-sharing transformers can bridge this gap by forming a "Generalization Circuit" during a prolonged "grokking" phase. A fundamental question arises: Is a grokked model superior to its non-grokked counterparts on downstream tasks? Furthermore, is the extensive computational cost of waiting for the grokking phase worthwhile? In this work, we conduct a mechanistic study to evaluate the Generalization Circuit's role in knowledge assimilation and transfer. We demonstrate that: (i) The inference paths established by non-grokked and grokked models for in-distribution compositional queries are identical. This suggests that the "Generalization Circuit" does not represent the sudden acquisition of a new reasoning paradigm. Instead, we argue that grokking is the process of integrating memorized atomic facts into an naturally established reasoning path. (ii) Achieving high accuracy on unseen cases after prolonged training and the formation of a certain reasoning path are not bound; they can occur independently under specific data regimes. (iii) Even a mature circuit exhibits limited transferability when integrating new knowledge, suggesting that "grokked" Transformers do not achieve a full mastery of compositional logic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。