用可学习的图记忆替代Transformer的前馈层,实现更透明的语言建模。
Graph Memory Transformer (GMT)

- 用记忆图替代传统前馈网络,通过重心间转移路由表示
- 模型仅8220万参数,无密集前馈层,训练稳定且可直接观察记忆流动
- 适合关注模型可解释性与架构创新的研究者
我们探究在仅解码器的Transformer中,是否可用显式学习的记忆图替代前馈网络(FFN)子层,同时保持自回归结构。提出的图记忆变压器(GMT)保留因果自注意力,但将每个标记的FFN变换替换为一个记忆单元,该单元通过学习的有向转移矩阵连接一组学习的中心点,实现标记表示的路由。在研究的base GMT v7实例中,每个16层变压器块包含128个中心点、128×128的边矩阵、引力源路由、标记条件的目标选择和门控位移读出。该单元返回从估计的源记忆状态到目标记忆状态的移动,而非检索值。最终模型为完全解码器的语言模型,含8220万可训练参数,无密集前馈层,相比评估中使用的10300万参数密集型GPT基线更小。该模型训练稳定,中心点使用、转移结构和源到目标移动均可直接观察。其验证损失和困惑度(3.5995/36.58)略高于更大规模的基线(3.2903/26.85),但在零样本基准测试中表现接近。这些结果并非追求最先进性能,而是支持以图引导的记忆导航替代密集内部变换的可行性与结构可解释性。更广泛的扩展、优化内核和更全面的评估留待后续工作。
原文摘要 · Abstract (English)
We investigate whether the Feed-Forward Network (FFN) sublayer in a decoder-only transformer can be replaced by an explicit learned memory graph while preserving the surrounding autoregressive architecture. The proposed Graph Memory Transformer (GMT) keeps causal self-attention intact, but replaces the usual per-token FFN transformation with a memory cell that routes token representations over a learned bank of centroids connected by a learned directed transition matrix. In the base GMT v7 instantiation studied here, each of 16 transformer blocks contains 128 centroids, a 128 * 128 edge matrix, gravitational source routing, token-conditioned target selection, and a gated displacement readout. The cell therefore returns movement from an estimated source memory state toward a target memory state, rather than a retrieved value. The resulting model is a fully decoder-only language model with 82.2M trainable parameters and no dense FFN sublayers, compared with a 103.0M-parameter dense GPT-style baseline used in the evaluation. The base v7 model trains stably and exposes centroid usage, transition structure, and source-to-target movement as directly inspectable quantities of the forward computation. It remains behind the larger dense baseline in validation loss and perplexity (3.5995/36.58 vs. 3.2903/26.85), while showing close zero-shot benchmark behavior under the evaluated setting. These results are not intended as a state-of-the-art claim; they support the viability and structural interpretability of replacing dense within-token transformation with graph-mediated memory navigation. Broader scaling, optimized kernels, and more extensive benchmark evaluation are left for subsequent work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。