arXiv:2501.05460cs.DCcs.AI2025-01ICML被引 23

将多模态模型的编码、预填充、解码分离,大幅降低内存占用并提升响应速度。

Efficiently Serving Large Multimodal Models Using EPD Disaggregation

  • 把编码、预填充和解码分到不同资源上运行,实现高效解耦。
  • 峰值内存降低15倍,单请求可处理10倍图像,批量大小提升22倍。
  • 适合需要高并发、低延迟的多模态服务部署场景。

大型多模态模型(LMMs)通过处理图像、音频、视频等多样化输入扩展了大语言模型(LLMs)的能力,但引入了多模态编码阶段,导致计算与内存开销显著增加,影响关键服务指标如首次标记时间(TTFT)和每输出标记时间(TPOT)。我们提出编码-预填充-解码(EPD)解耦框架,将编码、预填充和解码阶段分别部署在专用资源上。不同于现有系统将编码与预填充捆绑的做法,该方法解耦各阶段,带来新优化机会:包括多媒体标记缓存机制以实现高效传输、请求内编码负载并行化的新方式、针对解耦服务的最优资源分配模块,以及应对动态工作负载的角色切换机制。在主流LMMs上的实验表明,该方案在内存效率上提升达15倍(峰值内存降低),批处理规模扩大22倍,单请求支持图像数量增加10倍,KV缓存容量扩大2.2倍。同时,服务目标达成率提升90%-100%,首次标记时间减少71%。代码已开源于https://github.com/vbdi/epdserve。

原文摘要 · Abstract (English)

Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such as images, audio, and video, but at the cost of adding a multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first token (TTFT) and time per output token (TPOT). We introduce Encode-Prefill-Decode (EPD) Disaggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedicated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportunities and optimizations. These include a mechanism to cache multimedia tokens for efficient transfer, a novel way to parallelize the encoding load within a request, a module for optimal resource allocation for disaggregated serving, and a novel role-switching method to handle changing workload characteristics. Experimental evaluations with popular LMMs show substantial gains in memory efficiency (up to 15x lower peak memory utilization), batch sizes (up to 22x larger), 10x more images per request, and 2.2x larger KV caches. Furthermore, it leads to significant improvements in SLO attainment (up to 90-100% improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at https://github.com/vbdi/epdserve.

多模态模型服务优化资源解耦延迟降低

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。