让大模型注意力更聪明：自动分配全连接和滑动窗口注意力
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
- 用可学习掩码动态决定每层注意力用哪种模式
- 在0.6B和1.7B模型上性能优于人工规则，精度损失<2%
- 适合追求高效推理的工程师，尤其关注注意力结构优化
混合架构结合全注意力（FA）与滑动窗口注意力（SWA）是高效大模型推理的有前景方向。现有方法多依赖人工规则或简单后处理启发式策略，且缺乏对注意力行为的深入分析。我们提出可控制稀疏性的混合注意力框架ConSA，能在用户指定的稀疏度目标下学习最优的FA/SWA分配。ConSA采用L0正则化学习二值掩码，决定每个注意力单元选择FA或SWA，同时通过增强拉格朗日约束，在层或键值头粒度上强制实现目标稀疏度。我们在两个规模分别为0.6B和1.7B的大模型上评估ConSA。学习到的分配策略一致优于基于规则的基线，键值头粒度分配明显优于层粒度；学习到的模式将SWA置于底层，将FA集中在连续中层块，与规则方法的均匀交错模式显著不同。该结构在不同模型规模、稀疏度和粒度下均保持稳定，揭示了内在注意力行为的精细谱系。
原文摘要 · Abstract (English)
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely on hand-crafted rules or simple post-hoc heuristics for FA/SWA allocation and offer limited analysis of the attention behaviors underlying these designs. We propose Controllable Sparsity in Hybrid Attention (ConSA), a framework that learns optimal FA/SWA assignment under a user-specified sparsity target. ConSA employs L0 regularization to learn binary masks selecting between FA and SWA for each attention unit, while an augmented Lagrangian constraint enforces the target sparsity at either layer or KV-head granularity. We evaluate ConSA on two LLMs at the 0.6B and 1.7B scales. Learned allocations consistently outperform rule-based baselines, with KV-head-wise allocation yielding clear gains over layer-wise allocation. The learned patterns place SWA in the bottom layers and concentrate FA into contiguous middle-layer blocks, diverging from evenly interleaved patterns in rule-based methods. This structure persists across model scales, sparsity levels, and allocation granularities, revealing a fine-grained spectrum of intrinsic attention behaviors that underlies the learned allocation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。