用可复用的KV状态代替重复预填充，大幅降低个性化大模型服务延迟。
InferScale: GPU-Native KV Injection for Personalized LLM Serving
- 将记忆事实的KV表示提前计算并存于GPU，服务时直接注入缓存。
- 检索预算增至50时，首次生成时间降低72%-79%，吞吐量提升3.7-4.5倍。
- 无需修改引擎或微调模型，适合高并发个性化大模型部署场景。
大语言模型日益采用持久化个性化上下文（如积累的记忆档案或长对话历史），这些内容在用户多次请求间共享。现有内存系统（如Mem0、MemGPT、Zep）需反复从记忆中检索并注入提示，导致每次请求都重复预填充相同内容，使首次生成时间（TTFT）随检索预算增加而上升。我们提出InferScale，一种基于GPU的原生内存系统，将重复预填充替换为可重用的KV状态。该系统预先计算每个记忆事实的KV表示，结合语义嵌入存储于GPU，在服务时检索相关事实并直接注入vLLM的分页缓存。为支持旋转位置编码下的动态组装记忆，引入分块旋转位置编码（Chunked RoPE），在注入时才应用位置信息。独立编码会丢失联合预填充中的跨事实上下文，因此提出上下文窗口编码（Context-Window Encoding），在编码时包含前序对话片段，但仅缓存目标事实的KV。InferScale通过vLLM的KV连接器实现，无需引擎修改或模型微调。在LoCoMo数据集上，三个开源模型表现：当检索数k=50时，TTFT降低72%-79%（3.6-4.8倍），准确率60.3%（对比Mem0的63.3%），并发负载下吞吐量提升3.7-4.5倍。可复用的KV状态使服务延迟与检索内容规模解耦，同时保持应用质量。
原文摘要 · Abstract (English)
Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。