提出LinearKV,让混合型大模型用单个缓存状态实现位置无关缓存。
LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs
- 采用解耦初始化,线性层仅需一个初始状态而非拼接多个
- 单状态缓存可恢复92%完整性能,比精确组合更优且更快
- 无需训练,兼容现有缓存策略,适合部署在长序列任务中
大模型服务正越来越多地依赖位置无关缓存(PIC)。现有PIC方法针对全注意力模型设计,依赖基于标记索引的键值(KV)缓存来匹配可复用的词块、拼接其键值,并选择性重计算少数标记以恢复跨块上下文。然而,混合型大模型将多数注意力层替换为线性递推,仅暴露固定大小的状态,不再有可拼接或局部修复的标记索引KV缓存。这引发一个问题:PIC能否适用于混合模型?需要什么条件?本文提出LinearKV,一种无需训练的混合型PIC框架。核心思想是解耦初始化:每个线性层将其匹配的K个局部状态映射为单一初始状态,而全注意力层仍按原方式拼接其KV。LinearKV与现有PIC方法兼容,可直接复用其标记选择和重计算逻辑。实验表明,仅需一个缓存状态作为线性层初始值即可。相比之下,另一工作HYPIC所采用的精确组合所有K个状态以还原完整前缀状态的方法既非必要,且在某些架构上反而有害。在三个混合模型和三种PIC选择器下对比发现,在两个GDN模型上两者性能持平,均恢复高达92%的完整质量;而在Mamba-2模型上,精确组合导致性能崩溃——例如在EPIC下仅恢复46.6%完整质量,而单状态初始化达到86.8%。单状态初始化还更高效,将首次生成时间降至0.46倍完整预填充时间,而精确组合额外增加5%-17%延迟。结果在LongBench QA和RULER(8K–32K序列长度)上均成立。
原文摘要 · Abstract (English)
LLM serving is increasingly accelerated by position-independent caching (PIC). Existing PIC methods, however, are built for full-attention models, where a token-indexed KV cache underlies its core operations: matching reusable token chunks, concatenating their KV entries, and selectively recomputing a few tokens to restore cross-chunk context. Hybrid LLMs break these primitives---they replace most attention layers with linear recurrences that expose only a fixed-size state, leaving no token-indexed KV to concatenate or to locally repair. This raises a natural question: can PIC benefit hybrid models, and what would it take? We present LinearKV, a training-free hybrid-PIC framework. Its key insight is a \emph{decoupled initialization}: each linear layer maps its $K$ matched local states to a single initial state, while full-attention layers concatenate their KV as before. LinearKV is therefore compatible with existing PIC methods, reusing their token selection and recomputation as-is. Under this framework, we find that a \emph{single cached state} suffices as the linear layer's initializer. The algebraically principled alternative---composing all $K$ cached states into the exact full-prefix state, as concurrent work HYPIC does---is unnecessary and, on some architectures, even harmful. We compare the two across three hybrid models and three PIC selectors. On the two GDN models the two tie, both recovering most of full quality (up to $92\%$); on the Mamba-2 model, exact composition instead collapses under every selector---under EPIC, for instance, it recovers only $46.6\%$ of full quality, versus $86.8\%$ for a single cached block initializer. A single state initializer is also cheaper, cutting time-to-first-token to $0.46\times$ full prefill versus a further $5$--$17\%$ overhead for exact composition; results hold across LongBench QA and RULER at 8K--32K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。