通过分析成功与失败案例中注意力行为,实现长文本缓存压缩的高效优化。
REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
- 构建注意力行为矩阵,同时分析成功与失败场景下的注意力模式。
- 在LongBench v2上达到与最强基线相当精度,仅需1/32的存储空间。
- 首次引入多行为分析框架,适合关注长上下文建模与内存优化的研究者。
大型语言模型序列长度增长带来了关键-值(KV)缓存的巨大挑战。现有最先进的缓存淘汰方法主要分析成功检索推理案例中注意力头的行为,常忽视失败案例中的偏见和分心等多样化行为,限制了利用异构头行为提升淘汰性能的潜力。受混淆矩阵启发,我们提出注意力行为矩阵,全面分析成功与失败场景下的注意力头行为。通过最大化信噪比——强化成功案例中的有效推理路径,抑制失败案例中的噪声(如偏见与分心),我们提出首个基于多行为分析的REtrieval-reAsoning and Logic-constructed(REAL)KV缓存淘汰方法。全面评估表明,REAL在多种模型和基准上表现卓越;特别地,在LongBench v2上,其精度与最强基线HeadKV-R2相当,但仅需其1/32的存储空间(图1)。本工作为从仅关注成功案例转向全面、故障感知的方法提供了新视角,推动长上下文建模范式演进。代码已开源:https://github.com/yonseicasl/REAL。
原文摘要 · Abstract (English)
The growing sequence length of large language models poses significant challenges for key-value (KV) caches. Existing state-of-the-art cache eviction methods primarily analyze the inference behavior of attention heads in successful retrieval-reasoning cases, often overlooking diverse behaviors in failure cases, such as bias and distraction. This oversight limits the potential to leverage heterogeneous head behaviors for improved eviction performance. Inspired by the confusion matrix, we introduce an Attention Behavior Matrix to comprehensively analyze attention head behaviors in both success and failure scenarios. By maximizing the signal-to-noise ratio -- strengthening valid reasoning pathways in success cases while inhibiting noise from bias and distraction in failure cases -- we propose REtrieval-reAsoning and Logic-constructed (REAL) KV cache eviction, the first method to leverage multi-behavior analysis. Comprehensive evaluations show that REAL achieves remarkable performance across various models and benchmarks; notably, on LongBench v2, it achieves comparable accuracy to the strongest baseline, HeadKV-R2, while requiring 32x less space (Figure 1). By offering a novel perspective on behavior analysis, we pave the way for a shift from success-only to comprehensive, failure-aware methods in long-context modeling. Our code is available at https://github.com/yonseicasl/REAL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。