解决多模态大模型推理中的延迟与资源浪费问题,提升系统吞吐量和响应速度。
Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
- 通过多GPU协作解码加速视频预处理,降低首帧延迟
- 解耦视觉编码与语言模型执行,消除阶段间阻塞,支持资源共享
- 实测可处理3倍请求量或满足更严苛的延迟要求,性能提升4.4倍
多模态大语言模型(MLLM)通过三阶段流程——多模态预处理、视觉编码和语言模型推理——实现视觉理解。但该流程引入显著系统瓶颈:首先,视频解码等多模态预处理常主导时延,现有基于CPU的解码严重限制吞吐,而现有基于GPU的方案侧重吞吐并行,难以满足低延迟需求;其次,视觉编码为独立计算密集型阶段,无法与语言模型的prefill或解码阶段合并批处理,导致阶段间阻塞,增加生成延迟。即使部署于独立GPU,各阶段也存在算力与内存资源利用率低下问题。为此,本文提出FlashCodec与UnifiedServe两种互补设计,共同优化端到端推理流程。FlashCodec通过跨GPU协作解码,降低视频解码延迟,同时保持高吞吐;UnifiedServe采用逻辑解耦但物理资源共享的机制,优化视觉到文本及推理阶段的执行,消除阶段间阻塞,最大化显卡利用率。通过精细调度与干扰最小化,整体框架可支持最高3.0×的请求量提升或实现1.5×更严格的SLO约束,相较当前最优系统,吞吐率最高提升4.4×。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) extend LLMs with visual understanding through a three-stage pipeline: multimodal preprocessing, vision encoding, and LLM inference. While these stages enhance capability, they introduce significant system bottlenecks. First, multimodal preprocessing-especially video decoding-often dominates Time-to-First-Token (TTFT). Most systems rely on CPU-based decoding, which severely limits throughput, while existing GPU-based approaches prioritize throughput-oriented parallelism and fail to meet the latency-sensitive requirements of MLLM inference. Second, the vision encoder is a standalone, compute-intensive stage that produces visual embeddings and cannot be co-batched with LLM prefill or decoding. This heterogeneity forces inter-stage blocking and increases token-generation latency. Even when deployed on separate GPUs, these stages underutilize available compute and memory resources, reducing overall utilization and constraining system throughput. To address these challenges, we present FlashCodec and UnifiedServe, two complementary designs that jointly optimize the end-to-end MLLM pipeline. FlashCodec accelerates the multimodal preprocessing stage through collaborative multi-GPU video decoding, reducing decoding latency while preserving high throughput. UnifiedServe optimizes the vision-to-text and inference stages using a logically decoupled their execution to eliminate inter-stage blocking, yet physically sharing GPU resources to maximize GPU system utilization. By carefully orchestrating execution across stages and minimizing interference, UnifiedServe Together, our proposed framework forms an end-to-end optimized stack that can serve up to 3.0$\times$ more requests or enforce 1.5$\times$ tighter SLOs, while achieving up to 4.4$\times$ higher throughput compared to state-of-the-art systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。