通过复用重复上下文块,显著加速长文本推理且不损失推理质量。
ContextPilot: Fast Long-Context Inference via Context Reuse
- 构建上下文索引识别跨轮次重复内容,实现高效缓存复用。
- 相比顶尖方法,预填充延迟降低最高达3倍,长上下文下推理质量还提升。
- 模块化设计易集成现有系统,适合需高并发长上下文的AI应用。
随着大语言模型在检索增强生成、智能体记忆层和多智能体协同等场景中广泛应用,长上下文推理已成为关键需求。然而,随着输入上下文变长,预填充延迟成为主要瓶颈。现有加速技术面临权衡:要么保持推理质量但缓存复用有限,要么提升复用率却损害质量。本文提出ContextPilot,引入上下文复用机制以加速长上下文推理。其核心是构建上下文索引,识别不同用户交互间(如多轮对话)的重叠内容块;结合上下文排序与去重技术最大化KV-cache复用;并通过精炼的上下文标注机制保障复用过程中的推理质量。系统采用模块化架构,可无缝集成至现有推理引擎。实验表明,相比当前最优方法,ContextPilot将预填充延迟最高降低3倍,且在长上下文场景下甚至能提升推理质量。项目已开源:https://github.com/EfficientContext/ContextPilot。
原文摘要 · Abstract (English)
AI applications increasingly depend on long-context inference, where LLMs consume substantial context to support stronger reasoning. Common examples include retrieval-augmented generation, agent memory layers, and multi-agent orchestration. As input contexts get longer, prefill latency becomes the main bottleneck. Yet today's prefill acceleration techniques face a trade-off: they either preserve reasoning quality but deliver little KV-cache reuse, or improve reuse at the cost of degraded reasoning quality. We present ContextPilot, a system that accelerates prefill by introducing context reuse as a new mechanism for faster long-context inference. ContextPilot introduces a context index to identify overlapping context blocks across LLM interactions (e.g., across users and turns). It further proposes context ordering and de-duplication techniques to maximize KV-cache reuse. To preserve reasoning quality under reuse, it introduces succinct context annotations that prevent quality degradation. Finally, ContextPilot is built around a modular architecture with a clean interface that integrates with existing inference engines. Extensive evaluation shows that ContextPilot reduces LLM prefill latency by up to $3\times{}$ compared to state-of-the-art methods while preserving reasoning quality. At longer context lengths, it can even improve reasoning quality. ContextPilot is open-sourced at: https://github.com/EfficientContext/ContextPilot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。