揭示混合序列模型中内容路由的瓶颈：必须通过成对比较才能精准选择关注项。
When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
- 路由需双向上下文表征与成对令牌比较，缺一不可。
- 仅用单向或非成对机制的路由精度普遍低于29%，最高仅90.9%。
- 结合双向Mamba与低秩投影，可实现99.7%精度且线性推理成本。
我们揭示了混合序列模型中的路由悖论：基于内容的路由——决定哪些标记值得昂贵注意力计算——必然需要成对计算，这一需求无法避免。通过对三个任务、多个规模（200K至1.4B参数）和15+路由机制的20余组受控实验，我们全面绘制了路由格局。所有实现高路由精度的系统均依赖成对令牌比较；所有避免成对计算的方法均失败：循环模型（Mamba-1.4B: 29%）、记忆库（12%）、赌徒机制（0.7-3.6%）、对比预训练（1.6%）及其他12种方法均集中在1-29%。路由需两个要素：(1) 带双向上下文的每个标记表征，(2) 成对令牌比较。双向Mamba（O(n)）+成对比较达到99.5%；将全成对路由器替换为秩-1投影后提升至99.7%。在冻结Pythia-1B上添加一个双向层即可恢复99.4%路由。六种不同O(n)预处理机制（双向Mamba、Perceiver诱导点、端到端训练的因果注意力、稀疏注意力、双向注意力、秩-1投影）均成功；全局平均池化（1.9%）和傅里叶混合（0.9%）失败。路由信号位于约34维隐空间，无法被余弦相似度捕捉。非学习索引（布隆过滤器：90.9%；BM25：82.7%）可绕过瓶颈用于精确/关键词匹配。结合O(n)双向Mamba与秩-1成对投影，可在线性推理成本下实现99.7%路由。
原文摘要 · Abstract (English)
We identify a routing paradox in hybrid sequence models: content-based routing - deciding which tokens deserve expensive attention - requires pairwise computation, and this requirement is inescapable. Through 20+ controlled experiments across three tasks, multiple scales (200K to 1.4B parameters), and 15+ routing mechanisms, we map the routing landscape exhaustively. Every system that achieves high routing precision does so through pairwise token comparison. Every mechanism that avoids pairwise computation fails: recurrent models (Mamba-1.4B: 29%), memory banks (12%), bandits (0.7-3.6%), contrastive pretraining (1.6%), and 12 other approaches all cluster at 1-29%. Routing needs two ingredients: (1) per-token representations with bidirectional context and (2) pairwise token comparison. Bidirectional Mamba (O(n)) + pairwise comparison achieves 99.5%; replacing the full pairwise router with rank-1 projection improves this to 99.7%. Adding one bidirectional layer to frozen Pythia-1B recovers 99.4% routing. Six different O(n) preprocessing mechanisms (bidirectional Mamba, Perceiver inducing points, causal attention with E2E training, sparse attention, bidirectional attention, rank-1 projection) all succeed; global mean pooling (1.9%) and Fourier mixing (0.9%) fail. The routing signal occupies a ~34-dimensional latent subspace, invisible to cosine similarity. Non-learned indices (Bloom filter: 90.9%; BM25: 82.7%) bypass the bottleneck for exact/keyword matching. Combining O(n) bidirectional Mamba with rank-1 pairwise projection yields 99.7% routing at linear inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。