提出MPIC系统，让多模态模型推理更快更省资源。
MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving
- 用位置无关的缓存技术，避免重复计算
- 实测响应时间减少54%，吞吐量翻倍
- 适合需要快速处理图文混合内容的场景
当前多模态大模型（MLLM）服务普遍采用上下文缓存技术加速推理，但仅复用提示词初始序列的键值（KV）缓存，即使前缀略有差异也需重新计算全部缓存，效率低下。尤其在图文交错及多模态检索增强生成场景中问题突出。本文提出一种位置无关的缓存方法，设计并实现名为MPIC的缓存系统，解决系统与算法层面挑战：接收多模态数据时将KV缓存存储于本地磁盘，并在推理阶段并行计算与加载。为缓解精度下降，引入集成复用与重计算机制。实验表明，相比现有缓存系统，MPIC可实现最高54%的响应时间降低和2倍吞吐量提升，且保持几乎无精度损失。
原文摘要 · Abstract (English)
The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merely reuses the Key-Value (KV) cache of the initial sequence of prompt, resulting in full KV cache recomputation even if the prefix differs slightly. This becomes particularly inefficient in the context of interleaved text and images, as well as multimodal retrieval-augmented generation. This paper proposes position-independent caching as a more effective approach for multimodal information management. We have designed and implemented a caching system, named MPIC, to address both system-level and algorithm-level challenges. MPIC stores the KV cache on local disks when receiving multimodal data, and calculates and loads the KV cache in parallel during inference. To mitigate accuracy degradation, we have incorporated the integrated reuse and recompute mechanism within the system. The experimental results demonstrate that MPIC can achieve up to 54\% reduction in response time and 2$\times$ improvement in throughput compared to existing context caching systems, while maintaining negligible or no accuracy loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。