arXiv:2607.19390cs.LG2026-07

通过正交化读取机制,显著提升模型在噪声下的记忆召回能力。

The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory

  • 采用五次牛顿-舒尔茨迭代正交化读取矩阵,重构学习过程
  • 使模型逃逸成功率提升六倍,且不受任务难度影响
  • 该方法可移除且不依赖架构,适用于训练可塑性研究

近期研究表明,在读取时对mLSTM记忆矩阵进行正交化(使用五次牛顿-舒尔茨迭代,通过训练实现),能显著提升噪声环境下的关联回忆性能。这一效果可复现,但并非记忆容量提升。训练过程呈现长期平台期后突然突破的特征,正交读取通过在平台期重构学习问题来促进突破。三个特性验证此机制:第一,必须自洽——精确递归最小二乘读取(Mesa层)可复现该效果,而直通半元、增量写入规则、冻结随机键和普通归一化均失败;第二,具有均匀性——在学习率×难度网格上,其将逃逸概率提升约六倍,且无明显难度依赖,拓宽了有效学习率区间;第三,可移除性——在推理阶段应用于失败模型无法挽救,而在突破触发条件下逐步退火后,仍可保持完整精度。大量文献报告的性能增益无需特殊架构:固定预算下的求解率反映逃逸概率,遵循热/噪声规律(学习率弹性+3.0,梯度噪声弹性-1.65),原词汇表96的实验实为大批次噪声条件而非容量限制。直接解码记忆状态显示,失败模型中约一半关联以线性可恢复形式存在:平台期本质是读出失败而非存储不足。两个结论超越干预本身:当前用于架构选择的回忆基准部分反映训练可行性;该系统是‘涌现’的完整仪器化模型,其中尖锐行为阈值由逐渐积累的结构经抑制度量所引发。

原文摘要 · Abstract (English)

A recent report finds that orthogonalizing the mLSTM memory matrix at read time (five Newton-Schulz iterations, trained through) substantially improves noisy associative recall. The effect replicates, but it is not a memory improvement. Training on this task is a long chance plateau followed by a sharp escape, and the orthogonalized read acts by re-conditioning the learning problem during the plateau. Three properties establish this. It must be self-consistent: an exact recursive least-squares read (the Mesa layer) reproduces it, while straight-through halves, delta-rule writes, frozen random keys, and plain normalization all fail. It is uniform: across a learning-rate x hardness grid it multiplies the escape hazard roughly six-fold with no detectable hardness dependence, widening the workable learning-rate corridor that narrows for the baseline. And it is removable: applied to failed models at inference it rescues none, and annealed away on an escape-triggered schedule it leaves numerically stock mLSTMs at full accuracy. Much of the published gain needs no architecture at all: solved-rate at a fixed budget measures escape hazard, which follows a heat/noise law (learning-rate elasticity +3.0, gradient-noise elasticity -1.65) under which the original vocab-96 result is a large-batch noise condition rather than a capacity one. Decoding the memory state directly shows failed models carry roughly half their associations in linearly recoverable form: the plateau is a readout failure over half-written storage. Two conclusions travel beyond the intervention: recall benchmarks used for architecture selection partly measure trainability, and the system is a fully instrumented model organism of "emergence," in which a sharp behavioral threshold demonstrably arises from a censored metric over gradually accumulating structure.

记忆机制正交化训练优化涌现现象

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。