通过可调注意力预算，让Transformer模型在不同计算成本下灵活切换性能。
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
- 基于预算的注意力门控机制，按需分配计算资源。
- 在AG News上实现82.1%准确率与1.28倍加速，预算0.50时效果显著。
- 适合需要多级性能-成本平衡的部署场景，如边缘设备推理。
Transformer通常只提供单一推理成本，而实际部署常需多个成本-质量平衡点。本文研究预算注意力分配（Budgeted Attention Allocation），一种基于请求注意力预算的单调头门控机制。密集预热对稳定性至关重要：在合成序列任务中，一个预算模型在0.303估算注意力成本下达到99.7%准确率，在0.504成本下达100.0%。在自定义词级Transformer的AG News数据集上，硬门控适配将软成本控制转化为可测量的单线程CPU速度，于预算0.50时实现82.1%准确率和1.28倍加速。在预训练BERT-Mini的AG News上，预算结构剪枝在预算0.50时达87.6%准确率与1.20倍加速；验证排序的零样本密集后处理基线为86.1%，一次恢复训练提升至87.9%。在DBpedia14上，BERT-Mini预算门控在精确预算0.50时达97.4%，优于密集全注意力的96.6%。静态固定预算门控与恢复的密集专家仍表现强劲。贡献不在于全面超越，而是证明单一可调控检查点在跨预算下可行，可实现注意力成本与准确率之间的权衡，并转化为小规模CPU基准上的结构化加速。
原文摘要 · Abstract (English)
Transformers usually expose one inference cost per trained model, while deployed systems often need multiple cost-quality operating points. We study Budgeted Attention Allocation, a monotone head-gating mechanism conditioned on a requested attention budget. Dense warm-starting is important for stability: on a robust synthetic sequence task, one budgeted model reaches 99.7% accuracy at 0.303 estimated attention cost and 100.0% accuracy at 0.504 cost. On held-out AG News with a custom word-level transformer, hard-gate adaptation turns soft cost control into measured single-thread CPU speed, reaching 82.1% accuracy with 1.28x speedup at budget 0.50. In pretrained BERT-Mini AG News, budgeted structural pruning reaches 87.6% accuracy with 1.20x speedup at budget 0.50; a validation-ranked zero-shot dense post-hoc structural baseline reaches 86.1%, and one recovery epoch raises that per-budget specialist to 87.9%. On DBpedia14, BERT-Mini budgeted gates reach 97.4% at exact budget 0.50 versus 96.6% for dense full attention. Static fixed-budget gates and recovered dense specialists remain strong. The contribution is therefore not universal dominance, but a reproducible feasibility study of one controllable checkpoint across budgets that can trade attention cost for accuracy and be converted into measured structural speedups on small CPU benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。