通过感知状态衰减特性,压缩注意力模型的缓存占用,提升推理效率。
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

- 基于模型权重分析状态衰减规律,识别可压缩的长期记忆单元
- 采用不规则布局压缩状态缓存,相同内存下将时延降低42.6%、吞吐提升68.4%
- 适用于支持张量并行的推理系统,兼容不同结构的线性注意力模型
混合线性注意力架构已扩展至大型开源模型,其性能接近全注意力但显著降低键值缓存增长。然而,其就地递归状态更新使缓存管理复杂:前缀复用需保存状态检查点,而完整存储会增加内存压力,导致更多淘汰和重复预填充。通过对Gated DeltaNet(GDN)和Kimi Delta Attention(KDA)的衰减结构分析,我们发现不同头与通道保留前缀信息的时间尺度差异显著,称为“保留时域”。该现象揭示了持久状态检查点的巨大压缩潜力。基于此,我们提出衰减感知状态压缩(DASC),从模型权重中推导保留时域,选择长时域状态单元,并将其打包为不规则状态检查点布局。为适配张量并行推理引擎,DASC进一步均衡各并行秩上的压缩检查点。重用时,被省略单元置零或从有限后缀刷新,代价为额外计算。在Kimi-Linear上,保守配置下,压缩率高达2.63倍,同时保持接近全缓存性能;固定内存预算下,平均首字耗时(TTFT)下降42.6%,输入吞吐提升68.4%。更高压缩比下,后缀刷新可恢复大部分精度损失,代价为额外重放计算。Qwen-GDN也表现出类似质量-效率趋势,证明DASC可从通道级的KDA推广至头级的GDN。
原文摘要 · Abstract (English)
Hybrid linear-attention architectures have recently scaled to large open-weight models, offering quality competitive with full attention while substantially reducing key/value (KV) cache growth. However, their in-place recurrent-state updates complicate cache management: prefix reuse requires state checkpoints alongside full-attention KV, while storing state checkpoints in full increases memory pressure, leading to more evictions and repeated prefill. By analyzing the decay structure of Gated DeltaNet (GDN) and Kimi Delta Attention (KDA), we find that different heads and channels retain prefix information over markedly different timescales, which we term \emph{retention horizons}. This variation suggests substantial compression potential in persistent state checkpoints. Building on this observation, we introduce \emph{Decay-Aware State Compression} (DASC), which derives retention horizons from model weights, selects long-horizon state units, and packs them into a ragged state checkpoint layout. To integrate efficiently with tensor-parallel inference engines, DASC furtherly balances compressed state checkpoints across TP ranks. On reuse, DASC either zero-fills omitted units or refreshes them from a bounded suffix with additional compute cost. Across retrieval and end-to-end reasoning benchmarks on Kimi-Linear, conservative DASC configurations remain close to full caching while compressing KDA recurrent state checkpoints by $2.63\times$. Under fixed state checkpoint memory budgets, the resulting capacity gains reduce mean Time to First Token (TTFT) by 42.6\% and improve input throughput by 68.4\%. At larger compression ratio, suffix refresh recovers much of the accuracy lost to more aggressive omission, at the cost of additional replay computation. Qwen with GDN exhibits a similar quality--efficiency trend, showing that DASC extends from channel-wise KDA to head-wise GDN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。