微调评分函数让小预算下缓存压缩更高效，实测优于复杂结构改进。
Minimal-Intervention KV Retention via Set-Conditioned Diversity
- 用贪心选址思想替代传统筛选，仅改评分函数
- 在64和128缓存预算下，两组实验显著优于基线
- 适合关注推理模型压缩与轻量级优化的研究者
小预算下的KV缓存压缩涉及表示方式、头路由、压缩频率、解码行为和预算内评分等多个设计维度。我们在长序列数学推理任务（MATH-500）上，对两个蒸馏推理模型（Qwen-7B 和 Llama-8B 变体的 DeepSeek-R1-Distill）在缓存预算 b ∈ {64, 128} 下，评估了七种机制，结果全部失败。随后提出 α：对 TriAttention 保留评分器进行单函数修改，将 argmax-top-k 替换为受 V 空间冗余惩罚控制的贪心设施选址选择，惩罚权重为 λ。通过预注册协议在冻结开发集上调优 λ，再在独立保留集上验证；当 λ = 0.5 时，α 在四个（模型，预算）组合中的两个达到伯恩斯坦校正显著性（Qwen b=128、Llama b=64），无任何组合显著负向，且预注册分支 A 成功触发。结果呈现不对称性：极简评分修改在该场景胜过更复杂的结构重构，而匹配内存、SymPy评分、保留集验证的协议成为揭示这一现象的关键证据标准。
原文摘要 · Abstract (English)
KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring. We study seven mechanisms across these five families under matched mean cache on long-form mathematical reasoning (MATH-500~\cite{hendrycks2021math}) with two distilled-reasoning models (Qwen-7B and Llama-8B variants of DeepSeek-R1-Distill~\cite{deepseek2025r1}) at budgets $b \in \{64, 128\}$. All seven were rejected. We then propose $α$, a one-function modification to the TriAttention~\cite{mao2026triattention} retention scorer that replaces argmax-top-$k$ with greedy facility-location-inspired selection under a V-space redundancy penalty controlled by a single weight $λ$. A pre-registered protocol tunes $λ$ on a frozen development split and confirms on a disjoint held-out split; with $λ= 0.5$, $α$ clears Bonferroni on two of the four (model, budget) cells (Qwen $b{=}128$ and Llama $b{=}64$), no cell is significantly negative, and the pre-registered Branch~A triggers. The finding is asymmetric: a minimal scoring modification beat heavier structural redesigns in this regime, and the combined matched-memory, sympy-graded, held-out confirmation protocol is the evidence standard that made the asymmetry visible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。