通过复用98%视觉特征，实现多模态推理速度提升16倍
VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference
- 利用视觉与语言模型的缓存复用，避免重复计算
- 仅需2%-5%新计算量，精度接近全重算
- 适合大规模多模态推理系统部署
本文提出VLCache，一种缓存复用框架，通过复用先前多模态输入的键值（KV）缓存和编码器缓存，消除相同输入重复出现时的高成本重算。不同于以往启发式方法，我们首次形式化识别累积重用误差效应，并有效最小化非前缀缓存重用误差。进一步分析模型各层重要性，提出动态分层重算策略，在精度与效率间取得平衡。实验表明，VLCache在精度与全重算相当的情况下，仅需2%-5%的令牌计算量，实现1.2倍至16倍的TTFT加速。我们在SGLang基础上实现了原型系统，显著提升实际部署中的推理速度。
原文摘要 · Abstract (English)
This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。