让大模型高效处理长文档，用可组合的缓存块减少重复计算。
Cartridges at Scale: Training Modular KV Caches over Large Document Collections
- 将文档集拆分为独立缓存块，动态混合训练提升可扩展性。
- 百万级文本处理中，准确率比单一大缓存高10-31点。
- 适合需要低延迟、少提示词的长文本问答系统使用。
大语言模型虽能处理长上下文，但预填充数百万令牌效率低下，因大量内容在查询间保持不变。Cartridges 通过将文档集提炼为可复用的键值（KV）缓存，消除预填充并保持准确性。其关键局限在于缓存结构单一且不可组合：将整个集合编码为单一 KV 块难以扩展，而孤立训练后直接混合会令性能降至接近随机。本文提出 Cartridges at Scale (CAS)，一种支持可扩展多缓存学习的训练框架，采用动态干扰物混合机制与内存高效的预算管理器，实现数百个每文档缓存的 GPU 与持久存储间轮换。该方法可扩展至超过百万令牌的集合，在相似令牌预算下较单一大缓存提升 10-31 点准确率。在高压缩比下，最优缓存准确率仅比完整上下文学习低 2-6 点。结合检索选择缓存时，CAS 在消耗 3-4 倍更少提示词的前提下，达到或超越传统 RAG 准确率。
原文摘要 · Abstract (English)
Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries. Cartridges address this by distilling document collections into reusable key-value (KV) caches that eliminate prefilling while preserving accuracy. A critical limitation of this approach is that cartridges are monolithic and non-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance. We introduce Cartridges at Scale (CAS), a training framework for scalable multi-cartridge learning with dynamic distractor mixing and a memory-efficient budget manager that rotates hundreds of per-document cartridges between GPU and persistent storage. Our approach scales to collections exceeding a million tokens, improving over a monolithic cartridge by 10-31 points at comparable token budgets. Oracle cartridge accuracy falls within 2-6 points of full in-context learning even at high compression. When paired with retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3-4x fewer prompt tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。