让每个词动态选最省算力的注意力方式，自动平衡效率与效果。
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
- 用贝叶斯元控制器为每个词分配最优注意力策略，按计算成本自动选择
- 实测算力消耗降至25.1%（对比基线59.3%），路由熵降低12.5个百分点
- 无需额外负载均衡损失，可自适应避免路由坍缩，适合高效推理场景
标准Transformer对所有词和位置统一使用单一注意力机制，忽略上下文差异与算力预算。本文提出Meta-Attention框架，通过贝叶斯元控制器动态将每个词路由至全软注意力、线性核注意力或滑动窗口局部注意力中最合适的策略。该控制器将机制选择建模为带计算感知狄利克雷先验的后验推断，路由权重由变分后验q(alpha | x_t; phi)输出，训练目标为联合优化任务性能与注意力开销的证据下界（ELBO）。该设计生成有依据的路由不确定性估计，控制软到硬路由的过渡，无需人工负载均衡损失即可缓解路由坍缩，并在可忽略开销下实现优于确定性或无先验学习路由的算力-性能权衡。在小型语言模型基准上的第一阶段实验验证核心预测：贝叶斯控制器在硬路由下投影归一化FLOP成本为25.1%，而无先验基线为59.3%（下降34.2个百分点），路由熵从55.8%降至43.3%（下降12.5个百分点），表明狄利克雷先验有效防止路由坍缩，而非贝叶斯模型则默认使用全注意力。本文给出贝叶斯架构、ELBO训练目标及首个PyTorch原型，验证前向传播正确性、后验多样性以及与无先验基线的受控消融。代码已开源：https://github.com/KFEAL/meta-attention
原文摘要 · Abstract (English)
Standard transformer architectures apply a single attention mechanism uniformly across all tokens and sequence positions, irrespective of local context or computational budget. We propose Meta-Attention, a framework that dynamically routes each token to the most appropriate attention strategy -- full softmax attention, linear (kernel) attention, or sliding-window local attention -- via a Bayesian Meta-Controller. Unlike prior routing approaches that use deterministic or prior-free learned routing, the Meta-Controller treats per-token mechanism selection as posterior inference under a compute-aware Dirichlet prior: routing weights are the output of an amortised variational posterior q(alpha | x_t; phi) trained with an Evidence Lower Bound (ELBO) objective that jointly encodes task performance and attention-mechanism cost. This design produces principled routing uncertainty estimates that govern the soft-to-hard routing transition, mitigates routing collapse without ad hoc load-balancing losses, and yields better compute-performance trade-offs than deterministic or prior-free learned routing at negligible overhead. Phase 1 empirical results on a Tiny LM benchmark confirm core predictions: the Bayesian controller's learned routing distribution implies a projected normalised FLOP cost of 25.1% under hard routing, vs. 59.3% for the prior-free baseline (-34.2 pp), and reduces routing entropy from 55.8% to 43.3% (-12.5 pp), demonstrating that the Dirichlet prior prevents routing collapse while the non-Bayesian model defaults to full attention. We present the Bayesian architecture, ELBO training objective, and a Phase 1 PyTorch prototype validating forward-pass correctness, posterior diversity, and a controlled ablation against a prior-free baseline. Code available at: https://github.com/KFEAL/meta-attention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。