预计算记忆在组合、重建和修正时存在成本与失效问题,需及时更新才能保持准确。
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
- 通过缓存压缩与热重建维持记忆最新状态
- 分段组装记忆会导致准确性下降,重建成本占全量准备的很大比例
- 特定表述的补丁信息可临时更新记忆,适合中间阶段使用
语言模型可通过预计算的记忆(即模型对文本内容的预先存储读取)来回答问题,这些记忆可在多次请求间重复使用,避免重复读取。本文研究了该方法在不同条件下的表现:当记忆由独立部分拼接而成时,其正确性会下降;只有通过频繁重建才能保持时效性,而重建成本在我们的测量中占全量准备的很大比例;此外,模型还会忽略伴随记忆提供的修正信息,尤其依赖于提问方式。若预计算记忆能并行服务、低成本重建,并能被实时新信息覆盖,则可有效避免反复输入上下文。对于处理多样化查询的部署系统而言,应根据原始数据更新频率来决定记忆重建周期。实验表明,对键值缓存进行热重建的压缩模型,以及以粘贴文本或注入缓存状态形式提供针对性更新,均有助于保持记忆的时效性,后者可作为重建之间的临时方案,同时我们量化了相关开销并指出了待解决问题。
原文摘要 · Abstract (English)
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。