将代理的注意力缓存持久化到磁盘，提升边缘设备多代理推理效率。
Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
- 用4比特量化将代理缓存持久化到磁盘，避免重复预填充计算。
- 在4K上下文下，首次响应时间最快提升136倍，内存容量提升4倍。
- 适合资源受限场景下的多代理系统，如移动设备或嵌入式部署。
边缘设备上的多代理大模型系统面临内存管理难题：设备内存不足以同时存储所有代理的键值（KV）缓存。在配备10.2 GB缓存预算的Apple M4 Pro上，仅能容纳3个代理在8K上下文长度下以FP16格式运行。一个包含10个代理的工作流必须频繁地逐出和重新加载缓存。若无持久化，每次逐出都会强制通过模型进行完整重预填充——在4K上下文下每个代理耗时15.7秒。本文提出将每个代理的KV缓存以4比特量化格式持久化至磁盘，并直接恢复至注意力层，从而消除冗余的O(n)预填充计算。系统由三部分组成：为每个代理提供隔离的Q4 KV缓存的块池（safetensors格式）、支持多个代理并发推理的批量化量化缓存（BatchQuantizedKVCache），以及跨阶段上下文注入机制，可在不重新计算的情况下累积对话阶段中的注意力状态。在三种架构上评估：Gemma 3 12B（密集型GQA，48层）、DeepSeek-Coder-V2-Lite 16B（MoE MLA，27层）、Llama 3.1 8B（密集型GQA，32层）。缓存恢复使首字生成时间最多提升136倍（Gemma：4K–32K下22–136倍；DeepSeek：4K–32K下11–76倍；Llama：4K–16K下24–111倍；1K下3–10倍）。Q4量化相比FP16可将4倍更多的代理上下文装入固定内存。使用实际Q4 KV缓存测得的困惑度：Gemma下降0.7%，Llama上升2.8%，DeepSeek上升3.0%。代码开源：https://github.com/yshk-mxim/agent-memory
原文摘要 · Abstract (English)
Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。