通过检测注意力动态稳定，跳过冗余计算，提升长文本预填充速度。
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
- 基于注意力层更新动态，自动识别可跳过的稳定词元。
- 在多语言和视觉任务上提速最高达4.3倍，精度零损失。
- 无需训练，兼容FlashAttention等高效算子，适合部署场景。
长上下文场景下，预填充计算成本是大语言模型（LLMs）和大模态模型（LMMs）的主要瓶颈。尽管词元剪枝可减少序列长度，但现有方法依赖启发式规则，与FlashAttention等硬件高效内核不兼容。本文观察到词元会逐渐趋于语义固定点，后续处理冗余。为此，提出无训练的Delta Attention Selective Halting（DASH）策略，通过监控自注意力机制的层间更新动态，选择性跳过已稳定的词元。大量实验表明，DASH在语言与视觉基准上均具泛化能力，显著加速预填充过程，同时保持模型精度与硬件效率。代码将发布于 https://github.com/verach3n/DASH.git。
原文摘要 · Abstract (English)
Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward \textit{semantic fixing points}, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。