发现压缩缓存键值可作共享检索路由，提升长文本推理效率
Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations
- Cartridge 键作为稳定路由，引导压缩语料检索
- 90%以上压缩效果来自值向量，键向量可大幅精简
- 新初始化方法加速收敛，适合长上下文模型优化
长上下文大模型推理的瓶颈在于线性增长的键值缓存。近期工作提出 Cartridges，通过离线计算训练出远小于完整文档所需的缓存（最多节省 40 倍内存）。本文首次对 Cartridge 的学习结构进行机制探索，提出：(1) 键作为稳定、可共享的检索路由，用于压缩语料；(2) 大部分压缩发生在值向量中。我们在多种任务、模型族和规模上验证了该路由理论，例如在不同任务间删除键向量仅导致轻微性能下降。此外，我们提出一种名为 Sampled Chunk Initialization (SCI) 的改进初始化方法，表明其可实现比文献中更快的 Cartridge 收敛速度。研究为未来 Cartridge 训练优化提供了实证基础，对进一步扩展至关重要。
原文摘要 · Abstract (English)
A bottleneck for long-context LLM inference is the linearly growing KV cache. Recent work has proposed Cartridges, an approach which leverages offline compute to train a much smaller KV cache than is typically required for a full document (up to 40x less memory usage at inference time). In this paper, we present the first mechanistic exploration of the learned Cartridge key-value cache structure. In particular, we propose that (1) Cartridge keys act as stable, shareable retrieval routers for the compressed corpora and (2) most of the learned compression occurs within the Cartridge value vectors. We present empirical evidence of our routing theory across tasks, model families, and model sizes; for example, we can ablate the learned Cartridge key vectors between tasks with little performance loss. Finally, we propose a slight improvement in initialization called Sampled Chunk Initialization (SCI). We suggest that SCI can lead to faster Cartridge convergence than previously demonstrated in the literature. Our findings lay the groundwork for broader empirical study of Cartridge training optimization which may be crucial for further scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。