提出量化KV缓存交接技术，提升边缘设备多智能体大模型推理效率。
QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs
- 采用分词级混合精度分配与自包含缓存表示，实现高效量化交接。
- 在8K上下文下，延迟降低至全重预填充的38.6%，提升显著。
- 适合资源受限的边缘部署场景，尤其适用于多智能体协同任务。
边缘设备上的多智能体大模型系统需要高效传递隐式上下文，但现有方案要么成本高昂（重预填充），要么需全精度传输。本文提出QKVShare框架，通过分词级混合精度分配、自包含CacheCard表示及兼容HuggingFace的缓存注入路径，实现量化KV缓存交接。在150个GSM8K问题上，基于Llama-3.1-8B-Instruct的实验表明：自适应量化在多次交接中保持竞争力，在深层跳跃、高预算场景下相较均匀量化优势明显；手稿延迟方面，相比全重预填充，从1K上下文的130.7毫秒对150.2毫秒，到8K上下文的397.1毫秒对1029.7毫秒，均有显著改善。阶段分析显示，当前延迟主要来自注入后生成阶段，而非缓存创建。结果表明，量化KV交接是边缘部署的重要方向，但也亟需更深入的控制器消融和公平运行时对比。
原文摘要 · Abstract (English)
Multi-agent LLM systems on edge devices need to hand off latent context efficiently, but the practical choices today are expensive re-prefill or full-precision KV transfer. We study QKVShare, a framework for quantized KV-cache handoff between agents that combines token-level mixed-precision allocation, a self-contained CacheCard representation, and a HuggingFace-compatible cache injection path. Our current results support a narrower but clearer story than the original draft: on 150 GSM8K problems with Llama-3.1-8B-Instruct, adaptive quantization remains competitive under repeated handoff and shows its clearest gains against uniform quantization in deeper-hop, higher budget settings; for handoff latency, the QKVShare path reduces TTFT relative to full re prefill at every tested context, from 130.7 ms vs. 150.2 ms at nominal 1K context to 397.1 ms vs. 1029.7 ms at nominal 8K context;. Stage timing shows that post-injection generation, not card creation, dominates the current QKVShare latency path. These results position quantized KV handoff as a promising on-device systems direction while also highlighting the need for stronger controller ablations and apples-to-apples runtime comparisons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。