用少量学习的枢纽点实现高效序列建模，大幅降低计算开销。
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
- 引入枢纽路由机制，将注意力复杂度从O(n²)降至O(nM)
- 在1024序列长度下训练速度提升约90倍，仅损失少量性能
- 适用于新模型设计，适合追求高效推理的研究者
我们提出HubRouter，一个可插拔模块，将O(n²)注意力层替换为O(nM)的枢纽引导路由，其中M << n为少量学习的枢纽标记。在两种全新架构中验证：一种类Jamba的混合模型和12层Transformer；对预训练模型的回溯改造为无效尝试。HubRouter采用编码-解码-打分-决策流水线：M个学习的枢纽与所有标记交叉注意力，标记通过投影生成路由指纹，打分头选择前k个标记，稀疏委员会仅关注选定子集。在三个场景中验证：(1) Hub-Jamba在序列长度1024时训练吞吐量提升约90倍（优化后缩至10-15倍），困惑度降低4.2%（200.2 vs 209.0，单种子）；(2) 替换25%注意力层达到最优困惑度（268.0 vs 282.4纯Transformer）；(3) Hub-GPT实现严格因果路由，困惑度211.5±0.4（3种子），比Jamba差约3点，为避免O(n²)计算付出可衡量代价。修复后，块大小C影响微弱；此前块大小优势实为双向委员会泄漏所致。多种子枢纽数扫描（~105次运行，M=1-32）显示M=8-14为稳定收敛区间（4-5/5种子）；M=6经正交正则化可恢复至5/5；M≥20则种子敏感性上升。配套论文arXiv:2603.20997（Basu, 2026）定义了路由诊断任务。代码与脚本将公开。
原文摘要 · Abstract (English)
We introduce HubRouter, a pluggable module that replaces O(n^2) attention layers with O(nM) hub-mediated routing, where M << n is a small number of learned hub tokens. We demonstrate it in two from-scratch architectures: a Jamba-style hybrid and a 12-layer Transformer; retrofit into pretrained models is a tested negative case. HubRouter implements an encode-decode-score-council pipeline: M learned hubs cross-attend to all tokens, tokens project against hubs for routing fingerprints, a score head selects top-k tokens, and a sparse council attends only to the selected subset. We validate HubRouter in three settings. (1) Hub-Jamba yields a nominal 4.2% PPL improvement (200.2 vs 209.0, single seed; possibly within seed noise) and up to ~90x training throughput at sequence length 1024 in matched PyTorch-native baselines; an optimised baseline would narrow this to ~10-15x. (2) Graduated replacement of 25% of Transformer attention layers gives the best perplexity in our matched-budget sweep (268.0 vs 282.4 pure Transformer). (3) Hub-GPT provides strictly causal routing, achieving PPL 211.5 +/- 0.4 over 3 seeds (post council-causal fix); approximately 3 PPL worse than Jamba's 208.5 +/- 0.7, a measurable quality cost for avoiding O(n^2) computation. Post-fix, chunk size C has little effect; the pre-fix chunk-size benefit was an artifact of a bidirectional-council leak we found in adversarial review. A multi-seed hub-count sweep (~105 runs across M=1-32) reveals M=8-14 as the reliably-converging sub-band (4-5/5 seeds); M=6 is rescued to 5/5 by orthogonal regularization, while M>=20 shows increasing seed sensitivity. Companion paper arXiv:2603.20997 (Basu, 2026) defines the routing diagnostic task. Code and scripts will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。