用压缩率选关键信息,零参数实现自适应长序列注意力。
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
- 通过块级压缩比识别非冗余内容,动态生成注意力掩码。
- 8K上下文下达1.71比特/字节,优于密集注意力和多种基线模型。
- 无需额外参数或特殊代码,适合长文本建模场景。
数据自适应的稀疏注意力掩码显著优于固定模式(如BigBird和Longformer),甚至在长序列上超越稠密注意力。现有自适应方法通常需要可学习参数、定制梯度估计器或专用CUDA内核。我们发现经典数据压缩可提供有效掩码信号且无需额外参数。通过计算每块的gzip压缩比,识别出非冗余内容块,并仅在这些块间进行长距离注意力路由。直观上,无法被gzip压缩的块包含局部重复无法预测的信息,因此是理想的长程注意力目标。由于压缩特性依赖输入,生成的稀疏掩码能动态适应内容,无需学习参数、辅助损失或定制内核。在9200万参数、8000上下文的PG-19字节级语言建模任务中,该方法达到1.71比特/字节(BPB),优于稠密注意力(2.89)、BigBird(2.34)、Longformer(3.21)及重实现的SBM-Transformer(3.38)——唯一使用可学习掩码的基线——最高提升1.67 BPB,且不增加参数。优势随序列长度增长,在4000上下文时对BigBird领先0.05 BPB,8000时达0.63 BPB,收敛速度提升3.3倍。
原文摘要 · Abstract (English)
Data-adaptive sparse attention masks substantially outperform fixed patterns (e.g., BigBird and Longformer) and can even exceed dense attention on long sequences. Existing adaptive approaches---including SBM-Transformer, Dynamic Mask Attention, and NSA---typically require additional learnable parameters, custom gradient estimators, or specialized CUDA kernels. We show that classical data compression provides an effective masking signal with \textbf{no additional parameters}. By computing per-block gzip compression ratios, we identify non-redundant content blocks and route long-range attention selectively through them. Intuitively, blocks that gzip cannot compress contain information not predictable from local repetition, making them natural long-range attention targets. Because the compression profile is input-dependent, the resulting sparse mask adapts dynamically to content without learned parameters, auxiliary losses, or custom kernels. On PG-19 byte-level language modeling at 92M parameters with 8K context, our method achieves 1.71 bits-per-byte (BPB), outperforming dense attention (2.89), BigBird (2.34), Longformer (3.21), and a reimplemented SBM-Transformer (3.38)---the only learned-mask baseline---by up to 1.67 BPB while adding no parameters. The advantage grows with sequence length, with the gap over BigBird widening from 0.05 BPB at 4K context to 0.63 BPB at 8K, while convergence is 3.3$\times$ faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。