提出动态稀疏预填充方法,显著加速长文本大模型推理。
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing

- 直接从注意力图结构识别路由决策,避免复杂计算开销。
- 解决长上下文噪声累积问题,提升检索任务性能最高达+28.0个百分点。
- 适合追求高效长文本推理的开发者与研究者使用。
长上下文大模型推理中的注意力预填充阶段计算量呈平方级增长,成为主要瓶颈。传统稀疏注意力方法依赖固定模式或离线分析,难以适应输入变化;近期动态方法虽能实时路由,但依赖间接代理,存在开销且忽略软最大值后的质量层级。本文提出CRISP(Cliff-awaRe Input-adaptive Sparse Prefilling),针对动态路由中的两个结构性问题:首先,发现路由决策可直接从代理注意力图结构读取,用C_struct替代JSD路由,消除池化矩阵乘法和后续KL散度开销;其次,理论证明后软最大值质量悬崖导致长上下文下累积背景噪声达O(n),CRISP通过基于噪声底限的汇点感知阈值规避此问题。在InfiniteBench、RULER和LongBench上,跨两种模型家族实测表明,CRISP为当前最强稀疏方法,检索类任务表现匹配甚至超越精确密集注意力,相较基线最高提升+28.0个百分点,在512k token下实现最高5.30倍注意力加速,主要归功于选择阶段的O(n)噪声消除并保持结构完整性。
原文摘要 · Abstract (English)
The attention prefilling phase of long-context LLM inference scales quadratically, making self-attention a severe computational bottleneck. Traditional sparse attention methods mitigate this through fixed patterns or offline profiling, but lack the flexibility to adapt to input-dependent attention structure. Recent dynamic methods address this by routing heads to sparse patterns in real-time, but rely on indirect routing proxies with overhead and budget allocation mechanisms that overlook the post-softmax mass hierarchy. We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm. First, we show that the routing decision can be read directly off the structure of the proxy attention map. We replace the Jensen-Shannon Divergence (JSD) routing with C_struct, a structural proxy that measures mass at Vertical-Slash compatible positions and reproduces JSD's routing decisions while eliminating both the pooled matmul and subsequent KL divergence overhead. Second, we formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts. CRISP navigates this via a sink-aware threshold grounded in the noise floor. Empirically, across InfiniteBench, RULER and LongBench on two model families, CRISP is the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy benchmarks, recovering up to +28.0 pp on retrieval tasks over baselines and achieving up to a 5.30x attention speedup at 512k tokens, driven primarily by our O(n) noise elimination during selection while preserving structural integrity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。