根据层间扰动传播效果分配缓存,提升长文本生成效率
RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

- 通过注入自适应扰动测量各层对输出影响,构建敏感度评估
- 在LongBench上相同缓存预算下性能优于现有方法
- 适合追求长序列推理效率的模型部署者
长上下文大模型推理受键值缓存内存限制,而有限缓存预算在各层间的分配仍具挑战。现有方法依赖层深度、注意力统计或表征变化等代理指标,但这些指标未反映扰动在各层间的传播效应,可能导致敏感层缓存不足而容错层过度分配。为此,我们提出RippleKV,通过估计每层值缓存扰动对最终预测分布的影响来分配缓存。RippleKV在每个层的值缓存中独立注入范数自适应扰动,并在小规模校准集上测量输出端诱导的KL散度。对响应结果取平均得到与模型相关的敏感度轮廓,该轮廓不随深度单调变化。随后通过归一化敏感度得分并应用指数映射,将敏感度转化为层间预算乘数。比例参数控制敏感与容忍层间的分配差异,最终归一化保持总缓存预算不变。在LongBench上的实验表明,当缓存预算相同时,RippleKV在所有评估的压缩方法中实现了最高平均性能。
原文摘要 · Abstract (English)
Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging. Existing methods rely on proxies such as layer depth, attention statistics, or representation change. These proxies do not measure how perturbations at each layer propagate to the output and may therefore cause sensitive layers to be underallocated while tolerant layers are overallocated. To address this issue, we propose RippleKV, which allocates cache across layers by estimating how perturbations to each layer's value cache affect the final predictive distribution. RippleKV independently injects norm-adaptive perturbations into each layer's value cache and measures the induced KL divergence at the model output over a small calibration set. Averaging these responses yields a sensitivity profile specific to the model that need not vary monotonically with depth. RippleKV then converts the sensitivity profile into layer budget multipliers by normalizing the sensitivity scores and applying an exponential mapping. A ratio parameter controls the allocation disparity between sensitive and tolerant layers, while a final normalization preserves the KV cache budget. Experiments on LongBench demonstrate that RippleKV achieves the highest average performance among the evaluated KV cache compression methods under matched cache budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。