通过细粒度分组路由，提升大模型推理时的键值缓存压缩效率。
Sub-Token Routing for KV Cache Compression
- 在保留的令牌内按子组划分值向量，只保留部分组以压缩缓存。
- 在相同缓存预算下，比纯令牌级压缩提升性能，尤其在小预算时更明显。
- 适用于长文本与多模态生成场景，适合追求高效推理的研究者。
Transformer 推理常需大量键值（KV）缓存，尤其在长上下文语言建模和多模态生成中。现有压缩方法通常通过选择、淘汰、量化或压缩缓存令牌，或在语言模型推理前减少视觉令牌序列来降低缓存开销。本文提出子令牌路由（sub-token routing），在保留的令牌内部引入更细粒度的控制维度。该方法将每个保留的值向量拆分为多个组，仅保留选定的组，而查询和键状态保持不变。该方法设计为在令牌级压缩之后使用：首先通过令牌压缩方法确定保留哪些令牌；随后子令牌路由对这些保留令牌内的值状态进行压缩。在匹配的 KV 缓存预算下实验表明，加入子令牌路由可显著提升大型语言模型（LLM）和视觉语言模型（VLM）的性能，涵盖 Quest on LLaMA-2-7B、Qwen2.5-7B，以及 FastV/VisionZip 在 LLaVA 与 Qwen-VL 模型上的表现。在较小的缓存预算下收益更大，表明当进一步移除令牌成本上升时，值组路由尤为有效。总体而言，令牌级压缩与子令牌路由提供了互补的缓存优化路径。
原文摘要 · Abstract (English)
Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation. Existing compression methods usually reduce cache cost by selecting, evicting, quantizing, or compressing cached tokens, or by reducing the visual-token sequence before language-model inference. We introduce sub-token routing, a KV-compression method that adds a finer control axis inside retained tokens. It splits each retained value vector into groups and keeps only selected groups, while leaving query and key states unchanged. The method is designed to work after token-level reduction. First, a token-reduction method determines which tokens are retained. Then, sub-token routing compresses the value states inside those retained tokens. Experiments under matched KV budgets show that adding sub-token routing improves token-level reduction performance in both LLM and VLM settings, including Quest on LLaMA-2-7B and Qwen2.5-7B, and FastV/VisionZip across LLaVA and Qwen-VL models. The gains are larger at smaller KV budgets, suggesting that value-group routing is especially useful when further token removal becomes costly. Overall, token-level reduction and sub-token routing provide complementary ways to reduce KV cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。