提出新型记忆网络,解决长序列建模中梯度崩溃问题。
Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

- 用相位旋转约束状态更新,保持梯度稳定
- 85槽分层记忆树实现近100%准确检索
- 小模型媲美大模型的长程依赖能力
长达十余年,显式记忆架构如神经图灵机因反向传播时梯度灾难始终难以应用于语言建模。本文提出相位记忆网络(PMNet),通过单位相位动力学与分层可学习锚点结构,从根本上解决记忆不稳定性问题。通过将循环状态更新限制在复数单位圆上的相位旋转,PMNet无需特殊初始化即可保持梯度范数稳定。我们在字节级控制环境下验证其有效性:在合成复制粘贴任务中,使用包含85槽的分层记忆树(=∑ₕ₌₁⁴4^(h-1)),实现跨超长时间距离的近乎100%精确检索,远超局部滑动窗口注意力的感受野。尽管仅119M参数、训练于188亿词元,其零样本长程鲁棒性可媲美三倍大的Mamba模型。消融与梯度分析表明,显式记忆失败本质是结构对齐问题,而PMNet有效克服该问题,为可扩展序列建模提供理论基础。
原文摘要 · Abstract (English)
For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with \textit{Phasor Memory Network} (PMNet), a novel architecture that structurally resolves memory volatility through \textit{Unitary Phasor Dynamics} and \textit{Hierarchical Learnable Anchors}. Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive \textit{85-slot hierarchical memory tree} ($=\sum^{4}_{h=1}4^{h-1}$) to achieve near 100\% exact retrieval across temporal distances that completely exceed the local sliding window attention's receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。