arXiv:2604.16883cs.LGcs.AI2026-04

通过识别注意力汇聚点,跳过无效计算,提升长文本生成效率。

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

论文配图:SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
图 1 · 摘自论文原文
  • 基于训练中形成的稳定汇聚点,动态跳过无意义计算。
  • 512K上下文下实现2.03倍加速,精度不降。
  • 无需重新训练,适配多种大模型与多模态任务。

在大语言模型和多模态模型的长上下文解码中,注意力机制因每步需从显存加载大量键值缓存数据而逐渐成为内存瓶颈。现有加速方法常依赖启发式剪枝,牺牲准确性;深层原因在于对注意力汇聚现象理解不足,如盲目保留高分词、视早期词为不可删锚点、或依赖启发式头路由。本文揭示,注意力汇聚现象对应训练过程中形成的稳定、可到达且误差可控的固定点。据此提出SinkRouter:一种无需训练的可选路由框架,通过检测汇聚信号跳过将产生近零输出的计算。为实现真实加速,开发了支持块级分支与Split-K并行的硬件感知Triton内核。在LongBench、InfiniteBench、CVBench、MileBench、MMVP等多样化长上下文基准上,使用Llama-3.1-8B、Llama-3.1-70B、Yi-9B-200K、LLaVA-1.5-7B、LLaVA-1.5-13B等模型进行评估。结果表明,该方法在保持竞争力精度的同时持续提升解码效率,512K上下文下最高达2.03倍加速。

原文摘要 · Abstract (English)

In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache data from GPU memory. Existing acceleration strategies often trade efficiency for accuracy by relying on heuristic pruning that may discard useful information. At a deeper level, they also tend to indiscriminately preserve all high-scoring tokens, treat early tokens as indispensable anchors, or rely on heuristic head routing, reflecting an insufficient mechanistic understanding of the attention sink phenomenon. In this paper, we show that the attention sink phenomenon corresponds to a stable, reachable, and error-controllable fixed point constructed during training. Based on this insight, we propose SinkRouter, a training-free selective routing framework that detects the sink signal and skips computations that would otherwise produce near-zero output. To translate this mechanism into real-world acceleration, we develop a hardware-aware Triton kernel with block-level branching and Split-K parallelism. We conduct extensive evaluations on a diverse suite of long-context benchmarks, including LongBench, InfiniteBench, CVBench, MileBench, and MMVP, using both text-only and multimodal backbones such as Llama-3.1-8B, Llama-3.1-70B, Yi-9B-200K, LLaVA-1.5-7B, and LLaVA-1.5-13B. Across these settings, SinkRouter consistently improves decoding efficiency while maintaining competitive accuracy, and reaches 2.03x speedup with a 512K context.

长文本生成注意力优化推理加速多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。