用轮转采样实现长文本高效注意力，速度提升2.4倍且性能损失小于1%。
RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
- 每步轮转查询位置，各头独立采样同时发现全局模式
- 计算量从O(L²)降至O(L²/S²)，仅需一半注意力块即达99%性能
- 适合长文本理解、视频分析等需要高速推理的场景
注意力机制的二次复杂度严重制约大模型处理长上下文的能力。现有动态稀疏注意力方法存在预处理依赖、无法全局评估、违反查询独立性或计算开销高等缺陷。本文提出RRAttention，通过头级轮转采样策略，在保持查询独立性的前提下实现高效全局模式发现，并支持步长级聚合。该方法将复杂度从O(L²)降至O(L²/S²)，采用自适应Top-τ选择实现最优稀疏性。在自然语言理解（HELMET）和多模态视频理解（Video-MME）任务上的实验表明，RRAttention在仅计算一半注意力块的情况下恢复超过99%的全注意力性能，在128K上下文长度下实现2.4倍加速，优于现有动态稀疏注意力方法。
原文摘要 · Abstract (English)
The quadratic complexity of attention mechanisms poses a critical bottleneck for large language models processing long contexts. While dynamic sparse attention methods offer input-adaptive efficiency, they face fundamental trade-offs: requiring preprocessing, lacking global evaluation, violating query independence, or incurring high computational overhead. We present RRAttention, a novel dynamic sparse attention method that simultaneously achieves all desirable properties through a head \underline{r}ound-\underline{r}obin (RR) sampling strategy. By rotating query sampling positions across attention heads within each stride, RRAttention maintains query independence while enabling efficient global pattern discovery with stride-level aggregation. Our method reduces complexity from $O(L^2)$ to $O(L^2/S^2)$ and employs adaptive Top-$τ$ selection for optimal sparsity. Extensive experiments on natural language understanding (HELMET) and multimodal video comprehension (Video-MME) demonstrate that RRAttention recovers over 99\% of full attention performance while computing only half of the attention blocks, achieving 2.4$\times$ speedup at 128K context length and outperforming existing dynamic sparse attention methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。