自动优化稀疏注意力超参数，让Transformer加速更高效。
Self-Tuning Sparse Attention: Multi-Fidelity Hyperparameter Optimization for Transformer Acceleration
- 用贝叶斯优化与二分搜索结合，自动找每层每头的最佳超参。
- 在Llama-2-7B上比网格搜索快3.4倍，少用8.8倍评估次数。
- 适合想快速部署稀疏注意力的工程师和研究者。
稀疏注意力机制有望突破长序列Transformer的二次复杂度瓶颈，但实际应用受限于超参数难以适配：不同层、不同模型间最优配置差异大，现有方法（如SpargeAttn）依赖人工网格搜索。本文提出AFBS-BO（自适应保真度二分搜索与贝叶斯优化），一个全自动框架，无需人工干预即可发现层与头级别的最优超参数。该混合算法结合贝叶斯优化的全局探索与二分搜索的局部精调，利用多保真度评估（跨序列长度）降低调优成本。在Llama-2-7B上，AFBS-BO将超参发现速度提升3.4倍，评估次数减少8.8倍，同时找到高稀疏性配置，性能超越现有稀疏注意力基线，并接近稠密注意力质量。本工作使稀疏注意力从手动调参的启发式方法转变为可自优化的通用加速组件，支持多种Transformer架构与应用场景的即插即用加速。
原文摘要 · Abstract (English)
Sparse attention mechanisms promise to break the quadratic bottleneck of long-context transformers, yet production adoption remains limited by a critical usability gap: optimal hyperparameters vary substantially across layers and models, and current methods (e.g., SpargeAttn) rely on manual grid search to identify them. We propose AFBS-BO (Adaptive Fidelity Binary Search with Bayesian Optimization), a fully automated framework that discovers optimal layer- and head-specific hyperparameters without human intervention. Our hybrid algorithm combines Bayesian Optimization for global exploration with binary search for local refinement, leveraging multi-fidelity evaluation across sequence lengths to reduce tuning cost. On Llama-2-7B, AFBS-BO accelerates hyperparameter discovery by 3.4x with 8.8x fewer evaluations than grid search, and identifies high-sparsity configurations that outperform existing sparse attention baselines while closely matching dense attention quality. By transforming sparse attention from a manually tuned heuristic into a self-optimizing primitive, AFBS-BO enables plug-and-play acceleration across diverse transformer architectures and domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。