重复共享访问可促进模型领悟,但编辑传播依赖可寻址记忆。
Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memory
- 通过循环与共享内存的组合实验,发现重复访问是领悟的关键机制。
- 有记忆的模型编辑传播率高达0.71-0.96,无记忆模型不足0.03。
- 可寻址记忆比循环计算更能支持编辑传播,适合研究可解释性与知识更新。
我们在一个受控的合成知识图谱问答场景中,使用2x2网格对比循环递归与共享内存访问的影响:密集型Transformer(Dense)、循环Transformer(Loop)、带共享内存的密集骨干(Dense+Mem)和带共享内存的循环骨干(LMC)。学习方面,两种重复共享访问路径均突破Dense在分布外(OOD)的领悟障碍,说明重复共享是行为规律而非特定架构。编辑方面,在单次局部事实编辑后测量2跳传播,有记忆的模型(LMC 0.78-0.92,Dense+Mem 0.71-0.96)传播显著,而无记忆模型(Loop 0.04-0.30,Dense 0.00-0.03)几乎无效。该差异由记忆存在与否决定,而非循环机制。值得注意的是,Dense+Mem无循环,却仍能传播,说明关键在于可寻址写入与重读的内存位置。即使将存储粒度从N=128降至N=13,传播衰减但记忆/无记忆分裂依然存在,表明精细粒度提升精度而非提供传播能力。结果表明,学习与编辑能力可分离:重复共享足于实现领悟,但编辑传播依赖于是否具备可寻址记忆。
原文摘要 · Abstract (English)
We study factual edit propagation in a controlled synthetic knowledge-graph QA setting using a 2x2 grid that crosses loop recurrence with shared-memory access: a dense transformer (Dense), a looped transformer (Loop), a dense backbone with shared memory (Dense+Mem), and a looped backbone with shared memory (loop-memory coupling, LMC). The two factors dissociate. For learning, both routes to repeated shared access -- looped recomputation and repeated memory rereading -- cross the out-of-distribution (OOD) grokking barrier that Dense fails, so repeated shared access is the behavioral regularity, not a specific architecture. For editing, the substrates split along a different axis: applying a single localized factual edit (conditioned on direct success) and measuring 2-hop propagation on a shared pre-edit-correct set, the edit propagates strongly in both memory-bearing cells (LMC 0.78-0.92, Dense+Mem 0.71-0.96) and only weakly in the memory-free ones (Loop 0.04-0.30, Dense 0.00-0.03). The split is along the memory axis, not the loop axis: every memory-bearing seed exceeds every memory-free seed, with no detectable difference between the two memory cells. Crucially Dense+Mem has no recurrence, so the propagating ingredient is an addressable site that an edit can write to and later computation rereads, not loop recomputation; Loop is at best a partial intermediate. The affordance survives coarsening the store (N=128 to N=13): propagation attenuates but the memory/no-memory split persists, so fine granularity buys precision rather than the affordance itself. These results dissociate learning competence from editing affordance -- repeated shared access suffices to grok, but edit propagation depends on whether the substrate exposes an addressable memory that the forward computation can write to and later reread, an affordance that loop recurrence provides only partially.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。