通过分层混合注意力机制，高效处理长文本生成中的缓存瓶颈。
GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
- 按层差异化设计：浅层用软最大值窗口，深层用线性递推，兼顾精度与效率。
- 在不降质前提下，显著降低长文本生成的端到端延迟。
- 适合需要高吞吐长文本生成的场景，如文档摘要、代码生成。
随着大语言模型上下文长度持续增长，解码阶段键值（KV）缓存带来的内存输入输出与计算开销成为主要吞吐瓶颈。为此，我们提出GLIDE——一种分层引导的混合注意力机制，将滑动窗口软最大值注意力与线性递推聚合策略相结合。该方法基于层间异质性：浅层对移除软最大值敏感，深层则存在冗余，可被线性替代。GLIDE引入分层自适应机制，每层动态平衡高效线性递推与可变大小的软最大值窗口。相比均匀混合方法，GLIDE非均匀压缩软最大值覆盖范围，在减少整体KV缓存输入输出的同时保留关键表达能力。实证评估显示，该方法在长上下文生成中实现更优性能-效率权衡，显著降低端到端延迟且不牺牲生成质量。
原文摘要 · Abstract (English)
As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck. To address this, we propose GLIDE, a Guided Layerwise Hybrid Attention that strategically integrates sliding-window softmax attention with linear recurrent aggregation. GLIDE is motivated by layer-wise heterogeneity: early layers exhibit high sensitivity to softmax removal, while deeper layers demonstrate redundancy and tolerate aggressive replacement by linear alternatives. Leveraging this insight, GLIDE introduces a layer-wise adaptive mechanism wherein each layer balances an efficient linear recurrence with a variable-sized softmax window. Unlike uniform hybrid approaches, GLIDE non-uniformly compresses the softmax footprint across the model, reducing aggregate KV cache I/O while preserving expressive power where most vital. Empirical evaluations demonstrate the GLIDE achieves superior performance-efficiency tradeoffs, reducing end-to-end latency for long-context generation without compromising quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。