用动态指针路由机制，让模型在保持高效的同时大幅扩展记忆容量。
Graph Machine: Towards Better Pretraining via Edges
- 用可微更新的边作为动态指针，实现稀疏但灵活的状态访问。
- 仅用4个检索令牌/注意力头，预训练损失仅轻微下降，性能接近全量模型。
- 适合需要大规模记忆和低延迟推理的场景，如大模型预训练。
我们提出图机器（Graph Machine, GM），一种维持 $O(n)$ 级状态规模并利用稀疏动态路由访问的架构。与固定大小状态或静态稀疏路由的方法不同，GM 在稀疏层中保持 $O(n)$ 复杂度，且不将可访问状态限制在 $O(1)$。GM 通过边——由类似指针追踪的推荐机制可微更新的指针对象——来实现状态访问。我们在 Qwen3-0.6B 中用 GM 稀疏层替换了 75% 的密集 Transformer 层，基于 157 亿个标记从头预训练。在每个稀疏层中，每条键值头仅检索 2 个令牌时，损失仅轻微下降；检索 4 个令牌时，最优模型损失略有改善。
原文摘要 · Abstract (English)
We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。