发现块稀疏注意力中局部性不等于可达性,提出修复方法
Locality Does Not Imply Reachability: Boundary Repair in Block-Sparse Causal Attention

- 用结构依赖集分析块稀疏注意力的可达性边界缺陷
- 理论证明目标仅能依赖同块前缀,准确率上限1/K
- 提出边界桥接注意力,零参数新增连接修复可达性
稀疏因果注意力通常基于序列局部性:近邻标记易于访问,远距离标记可被丢弃以降低计算成本。本文研究序列局部性与注意力图可达性之间的不匹配问题。在固定块因果注意力中,相邻标记在每一层深度都可能在注意力图中断开。通过结构依赖集形式化这一边界伪影:若每层使用相同固定块因果掩码且所有操作为位置独立,则目标表示仅依赖于其所在块的前缀。由此构建出K路边界复制分布,其顶1准确率上限为1/K,期望交叉熵下界为log K。进一步推导出相位条件覆盖函数,表明可达性同时依赖源-目标距离及目标在块中的偏移。这些覆盖规律可预测稀疏模式失效时机、修复有效性,以及滑动窗口注意力与边界修复不可互换的原因。边界桥接注意力作为构造性验证:保持固定块路径,通过共享投影在块边界附近添加零额外参数的辅助因果边。1024标记受控实验显示增益集中于覆盖对齐诊断。作为外部有效性证据,固定检查点8K标记Qwen2.5-7B探测器也呈现相同覆盖不兼容模式。贡献在于为块稀疏因果注意力中的局部性-可达性不匹配提供理论指导的诊断框架,包括相位条件覆盖分析与最小构造性修复。
原文摘要 · Abstract (English)
Sparse causal attention is usually described by sequence locality: nearby tokens should remain easy to access, while distant tokens may be dropped to reduce cost. This paper studies a mismatch between sequence locality and attention-graph reachability. In fixed block causal attention, two adjacent tokens can be disconnected in the attention graph at every depth. We formalize this boundary artifact through structural dependency sets: if every attention layer uses the same fixed block causal mask and all remaining operations are positionwise, a target representation can depend only on tokens in its own block prefix. This yields an architecture-level boundary-copy separation for a constructed K-way boundary-copy distribution, with top-1 accuracy upper bound 1/K and expected cross-entropy lower bound log K. We then derive phase-conditioned coverage functions showing that reachability depends on both source-target distance and the target's offset within its block. These coverage laws predict when a sparse pattern should fail, when a repair can help, and why sliding-window attention and boundary repair are not interchangeable. Boundary Bridge Attention is treated as a constructive witness: it preserves the fixed block path and adds zero-additional-parameter auxiliary causal edges near block boundaries using shared projections. Controlled 1024-token experiments show that gains concentrate in coverage-aligned diagnostics. As secondary external-validity evidence, a fixed-checkpoint 8K-token Qwen2.5-7B probe shows the same coverage-incomparability pattern. The contribution is a theory-guided diagnostic framework for locality-reachability mismatch in block-sparse causal attention, together with phase-conditioned coverage analysis and a minimal constructive repair.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。