提出首个针对长文本模型键值缓存压缩的诊断基准，精准定位压缩失败原因。
KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
- 构建25种压缩方法的分类体系，关联8个可验证实现与诊断指标。
- 在Qwen3-8B上覆盖59800次压缩实验，发现63.2%存在覆盖率不足问题。
- 提供可复现的失败案例，适合研究缓存压缩与注意力机制的学者使用。
KV缓存压缩能降低长上下文内存占用，但整体任务得分无法揭示哪些正确执行失败及原因。我们提出KVDiagnosis，一个诊断数据集与基准，包含三方面贡献：首先，建立25种方法的分类体系，将方法归入五类机制家族，并链接至八个经验证的实现及其有效诊断测量；其次，对每个支持的方法设置，固定分片下对比各源与全缓存（FullCache）控制，分别筛选出全缓存正确但压缩错误（C-to-W）的样本，确保无压缩器定义另一压缩器的测试集；第三，采用统一记录格式，关联成对输出、运行元数据与缓存、似然、注意力及解码测量，明确标注适用状态。在Qwen3-8B上，四个基于证据的工作负载共生成59,800次支持的压缩运行，覆盖2,600个源和12,520个C-to-W行。在固定诊断规则下，63.2%的案例具有低或部分测量/预测覆盖率。仅19行（0.2%）同时具备高测量/预测覆盖率与强似然漂移；另有2,126行（17.0%）保留结构位置可寻址性，其表示保真度未知但表现出相同漂移。相比成功压缩对照组（C-to-C），所有十项诊断均能有效区分失败与成功压缩（分层AUROC 0.684–0.871）。在96个可复现的低误差率（EAR）失败中，受控的4倍证据-注意力增强修复了29.2%，远超计数匹配的假干预（6.3%）与匹配的C-to-C对照组降级（3.3%）。代码与数据见 https://github.com/ChosenQC/KVDiagnosis。
原文摘要 · Abstract (English)
KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-method taxonomy groups methods into five mechanism families and links them to eight verified implementations and their valid diagnostic measurements. Second, for every supported method setting, we evaluate all sources in each fixed split against a per-source FullCache control before selecting FullCache-correct/compressed-wrong (C-to-W) rows separately for each method-setting, so no compressor defines another's test set. Third, a common record format links paired outputs and run metadata to cache, likelihood, attention, and decoding measurements with explicit applicability states. On Qwen3-8B, four evidence-aware workloads yield 59 800 supported compressed runs over 2600 sources and 12 520 C-to-W rows. Under fixed diagnostic rules, 63.2% have low or partial measured/projected coverage. Only 19 rows (0.2%) combine high measured/projected coverage with strong likelihood drift; another 2,126 (17.0%) preserve structural position addressability, for which representation fidelity remains unknown, while showing the same drift. Against C-to-C success controls, all ten diagnostics separate failed from successful compression (stratified AUROC 0.684-0.871). Among 96 reproducible low-EAR failures, a controlled 4x evidence-attention boost repairs 29.2%, versus 6.3% under a count-matched sham intervention and 3.3% degradation on matched C-to-C controls. Code and data are available at https://github.com/ChosenQC/KVDiagnosis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。