用稀疏注意力提升长文本模型效率，实测32K上下文速度提升10.9倍
BF1: A Causal Dyadic Sparse-Attention Retrofit for Efficient Long-Context Transformers
- 设计块对齐的稀疏注意力结构，结合局部、全局和对数间隔历史块
- 32K上下文时每层推理速度比稠密注意力快10.91倍，8层改造后首词延迟降15.3%
- 适合需要高效长序列建模的部署场景，尤其对资源受限环境友好
稠密因果注意力在长上下文场景下依然计算昂贵。本文提出BF1，一种确定性的分块对齐双线性稀疏注意力机制，结合小范围精确局部邻域、全局首块及对数间距的历史块。该结构关联于已有对数稀疏与膨胀注意力模式；本文贡献包括正确性保障的预训练模型改造、拓扑控制对比研究及系统性能分析，揭示了每层稀疏度与整模型延迟的关系。固定块宽下，每层仅需O(n log n)个选中交互，图通信深度为O(log n)。在NVIDIA RTX PRO 6000 Blackwell GPU上，优化的BF16实现于2K至4K token间超越稠密注意力，32K时每层预填充速度提升10.91倍。改造Qwen3-0.6B模型中8/28个注意力层，使8K、16K、32K上下文下的暖启动首词时间分别降低7.7%、11.3%、15.3%，其余稠密层保持渐近二次复杂度。在匹配的1,000步、16.384M token适配协议下，BF1在三个种子上均排名第一：平均困惑度1.68639，优于静态随机非局部图（1.69154）、稠密继续训练（1.69258）和等预算局部滑动（1.81505）。在种子1234下，密集继续训练与静态随机图分别以0.3169–0.4055%和0.2441–0.3642%优势超过BF1。结果证实BF1是可复现的稀疏算子与选择性改造原语，在实际长上下文系统中具备价值。论文评估了数值正确性、选中交互扩展性、内核性能、部分模型推理与匹配的下一个词语言建模。
原文摘要 · Abstract (English)
Dense causal attention remains expensive at long context even when implemented with highly optimized exact kernels. We study BF1, a deterministic block-aligned dyadic sparse-attention route that combines a small exact local neighborhood, a global first block, and logarithmically spaced historical blocks. The route is related to prior log-sparse and dilated attention patterns; our contribution is a correctness-gated pretrained-model retrofit, a matched topology-control study, and a systems characterization that connects per-layer sparsity to whole-model latency. For fixed block width, every converted layer uses O(n log n) selected token interactions and has O(log n) graph communication depth. On an NVIDIA RTX PRO 6000 Blackwell GPU, an optimized BF16 implementation crosses dense attention between 2K and 4K tokens and reaches a 10.91x per-layer prefill speedup at 32K. Retrofitting eight of 28 Qwen3-0.6B attention layers lowers warm whole-model time to first token by 7.7%, 11.3%, and 15.3% at 8K, 16K, and 32K, respectively, while the remaining dense layers keep the complete model asymptotically quadratic. Under a matched 1,000-step, 16.384M-token adaptation protocol, BF1 ranks first across three training seeds: mean report perplexity is 1.68639 versus 1.69154 for a matched static-random nonlocal graph, 1.69258 for dense continued training, and 1.81505 for equal-budget local sliding. At seed 1234, the packed-report paired interval places Dense-CT 0.3169-0.4055% above BF1 and static-random graph 17 0.2441-0.3642% above BF1. These results establish BF1 as a reproducible sparse operator and selective retrofit primitive with real long-context systems value. This paper evaluates numerical correctness, selected-interaction scaling, kernel performance, partial-model inference, and matched next-token language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。