动态注意力掩码让大模型长文本推理更快更准
DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
- 自适应生成注意力掩码，不依赖预设结构或微调
- 在长序列上接近全注意力效果，计算量大幅降低
- 适合需要高效处理长文本的LLM部署场景
长上下文理解对众多NLP应用至关重要，但变换器因自注意力的二次复杂度导致效率低下。稀疏注意力方法虽能缓解开销，却多采用静态预设掩码，无法捕捉异质注意力模式，导致令牌间交互不佳，限制了长序列任务中的适应性与检索精度。本文提出一种动态稀疏注意力机制，在注意力图层面自适应生成掩码，保留各层与头间的异质模式。相比现有方法，无需微调或预定义掩码结构，同时保持计算高效。通过学习上下文感知的注意力结构，该方法与全注意力模型高度对齐，实现最小性能损失的同时显著降低内存与计算开销。DAM为大规模语言模型的实际部署提供可扩展替代方案，不牺牲检索性能。
原文摘要 · Abstract (English)
Long-context understanding is crucial for many NLP applications, yet transformers struggle with efficiency due to the quadratic complexity of self-attention. Sparse attention methods alleviate this cost but often impose static, predefined masks, failing to capture heterogeneous attention patterns. This results in suboptimal token interactions, limiting adaptability and retrieval accuracy in long-sequence tasks. This work introduces a dynamic sparse attention mechanism that assigns adaptive masks at the attention-map level, preserving heterogeneous patterns across layers and heads. Unlike existing approaches, our method eliminates the need for fine-tuning and predefined mask structures while maintaining computational efficiency. By learning context-aware attention structures, it achieves high alignment with full-attention models, ensuring minimal performance degradation while reducing memory and compute overhead. This approach provides a scalable alternative to full attention, enabling the practical deployment of large-scale Large Language Models (LLMs) without sacrificing retrieval performance. DAM is available at: https://github.com/HanzhiZhang-Ulrica/DAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。