提出稀疏令牌路由机制，让Transformer动态选择计算资源
Sparse Token Routing in Efficient Transformers
- 设计双流Transformer，通过学习门控机制决定令牌处理方式
- 门控信号准确区分重要与非重要令牌（p<10^-10）
- 在不损失精度前提下显著降低计算量，适合高效部署
高效Transformer研究常主张并非所有令牌都需要同等计算量。本文通过SEWN——一种双流Transformer，利用学习得到的门控机制将令牌路由至轻量或全容量处理路径，对这一主张进行端到端验证。实验表明，与参数匹配的基线相比，路由机制引入的精度变化可忽略不计；而门控的令牌重要性信号是否有效，关键取决于其学习方式。静态词典初始化的先验在BoolQ数据集上无法通过反事实忠实性测试，而完全上下文感知的门控在两个评估任务中均实现极显著的分离（p<10^-10），且未改变任务精度。
原文摘要 · Abstract (English)
Efficient-transformer research often motivates token pruning and adaptive computation with the claim that not all tokens require equal computational effort. We test this claim end to end using SEWN, a two-stream Transformer that routes tokens through either lightweight or full-capacity processing using a learned gate. Across our experiments, routing introduces negligible accuracy change relative to parameter-matched baselines, while the gate's token-importance signal depends critically on how it is learned. A static lexicon-seeded prior fails a counterfactual faithfulness test on BoolQ, whereas a fully contextual gate achieves highly significant separation ($p<10^{-10}$) on both evaluated tasks without changing task accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。