arXiv:2601.16531cs.LGcs.AI2026-01被引 1

研究高频键冲突对记忆模型训练的影响,发现消除冲突反而让模型表现变差。

A Collision-Free Hot-Tier Extension for Engram-Style Conditional Memory: A Controlled Study of Training Dynamics

  • 用最小完美哈希函数构建无冲突热层,保留原冷层对比实验
  • 训练中热区损失先低后高,冷区最终反超,且无冲突配置翻转更早
  • 冲突带来的噪声或具正则化作用,当前模型瓶颈在门控分配而非索引精度

我们探究高频键冲突是否是恩格拉姆式条件记忆的主要瓶颈。为隔离冲突影响,提出Engram-Nine:通过最小完美哈希函数映射最频繁n-gram构成无冲突热层,同时保留原始多头哈希查找作为冷层。在严格参数一致条件下,无冲突设计并未持续降低验证损失。通过路径分层评估(将每标记损失分解为热/冷贡献),发现训练过程中存在稳定的“热到冷优势翻转”现象:热区(高频)初始损失较低,但冷区最终超越。关键的是,无冲突配置的翻转时间早于有冲突基线,表明冲突具有隐式正则化作用。我们还发现门控错配:门控早期学习偏好热区,但该偏好在翻转后仍持续,导致更高损失位置获得更高权重。结果表明,仅提升查找精度无法保证更好训练效果,主要瓶颈可能在于门控信用分配而非索引准确率,冲突引入的噪声或具有益正则化,不应被盲目消除。

原文摘要 · Abstract (English)

We investigate whether high-frequency key collisions are a primary bottleneck in Engram-style conditional memory. To isolate the effect of collisions, we introduce Engram-Nine, a collision-free hot-tier extension that maps the most frequent n-grams through a Minimal Perfect Hash Function (MPHF) while retaining the original multi-head hashed lookup as a cold tier. Under a strictly iso-parameter setup, the collision-free design does not consistently improve validation loss. Through route-stratified evaluation (decomposing per-token loss into hot/cold contributions), we uncover a consistent "hot-to-cold advantage flip" during training: hot (high-frequency) positions initially have lower loss, but cold positions eventually surpass them. Crucially, collision-free configurations flip earlier than collision-prone baselines, suggesting that collisions act as implicit regularization. We also identify a gating mismatch: the gate learns to favor hot positions early in training, but this preference persists even after the flip, assigning higher weights to positions with higher loss. Our findings suggest that improving lookup precision alone does not guarantee better training outcomes. The dominant limitation may lie in gating credit assignment rather than index accuracy, and collision-induced noise may provide beneficial regularization that should not be naively eliminated.

记忆网络门控机制正则化哈希优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。