Engram在图像生成中并非内容检索，而是起辅助路径作用。
Does Engram Do Memory Retrieval in Autoregressive Image Generation?
- 用二维n-gram哈希注入视觉Transformer，实现快速增量推理。
- 所有增强版本FID均高于基线，说明未提升生成质量。
- 门控机制和随机表实验表明其本质是残差路径而非内容检索。
Engram模块——一种基于哈希键的O(1)关联记忆——被证明可提升大语言模型预训练效果，其解释为对重复局部标记模式的直接内容寻址。我们探究该机制是否适用于自回归（AR）图像生成。将Engram模块适配至视觉任务，采用2D空间n-gram哈希、门控融合及兼容KV缓存的增量推理，并注入到类条件AR生成器（在ImageNet 256x256上训练）。在骨干网络与记忆预算比ρ∈[0.17, 0.90]的全范围测试中，所有含Engram的变体均劣于纯AR基线，表明其仅节省骨干计算量，未能独立提升样本质量。进一步探查发现：完全禁用Engram路径导致灾难性退化；而极小恒定门控（g=0.10）表现与学习门控相当甚至更优，不符合强内容寻址机制。捐赠-探测实验显示，替换哈希输入为匹配、对抗或同类别随机样本，下一标记分布无统计差异；而表结构坍缩或随机化则使性能下降二至三个数量级。最后，从零开始训练时冻结整个记忆表为标准正态噪声（𝒩(0,1)），仅导致ΔFID=0.10，且反而提升Inception Score。综合表明，Engram在自回归图像生成中并非内容寻址检索器，而是一个受门控的架构旁路：其收益主要来自路径本身，学习表仅贡献微弱分布修正。
原文摘要 · Abstract (English)
The Engram module -- a hash-keyed, O(1) associative memory injected into Transformer layers -- was recently shown to improve large language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-memory budget ratios $ρ{\in}[0.17, 0.90]$, every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate -- inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire memory table frozen to $\mathcal{N}(0, 1)$ noise costs only $Δ\text{FID}{=}0.10$ and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。