通过交替使用局部与全局注意力，提升长序列建模效果。
Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
- 交替使用滑动窗口与压缩/选择性注意力，增强长距离依赖传播。
- 相比NSA减少50%键值缓存内存，提升常识推理与长文本理解能力。
- 适用于340M至1.3B参数模型，适合长序列任务场景。
本文系统分析了原生稀疏注意力（NSA），提出针对性改进以增强长上下文建模能力。核心发现是：在不同层间交替使用局部（滑动窗口）和全局（压缩、选择性）注意力，而非固定模式，能更有效地传递长距离依赖，显著提升长序列任务表现。同时，进一步优化NSA分支：滑动窗口分支引入多头潜在注意力（MLA），压缩与选择性分支采用分组头潜在注意力（GLA）。该方法使键值缓存内存减少50%的同时，提升了模型的常识推理与长文本理解能力。在340M至1.3B参数模型（分别在150亿与1000亿标记符上训练）上的实验表明，本方法在常识推理与长上下文理解任务中达到或超越全注意力与原生稀疏注意力表现。
原文摘要 · Abstract (English)
In this work, we conduct a systematic analysis of Native Sparse Attention (NSA) and propose targeted improvements that enhance long-context modeling. A key insight is that alternating between local (sliding-window) and global (compression, selective) attention across layers, rather than using fixed patterns, enables more effective propagation of long-range dependencies and substantially boosts performance on long-sequence tasks. Meanwhile, we further refine NSA's branches with Latent Attention that the sliding-window branch is enhanced with Multi-head Latent Attention (MLA) while compression and selective branches adopt Group-head Latent Attention (GLA). These changes reduce KV-cache memory by 50\% versus NSA while improving the model's common-sense reasoning and long-text understanding capabilities. Experiments on models from 340M to 1.3B parameters (trained on 15B and 100B tokens) show our method matches or exceeds full attention and native sparse attention in both common-sense reasoning and long-context understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。