针对多模态模型服务的复杂性，提出动态调度系统以提升吞吐并降本。
ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
- 将多模态推理流程拆解为可独立优化的模块，按需弹性扩展。
- 在128张GPU集群上实现3.3-5.5倍吞吐提升，成本降低25%-41.3%。
- 适合需要高并发、低延迟且多模态输入的生产级模型部署场景。
大型多模态模型（LMMs）在理解图像、视频和音频方面表现出色，超越了文本处理能力。然而，其复杂架构及多阶段推理流水线中各阶段的异构特性，使在生产环境中高效服务这些模型面临巨大挑战。本文首次对两种主流LMM架构——仅解码器型与交叉注意力型——在六种代表性开源模型上进行系统分析，揭示关键系统设计启示。同时，对生产环境中的LMM推理轨迹进行深入分析，发现请求分布具有可变性和重尾特征，且存在突发流量模式。基于上述洞察，我们提出ModServe，一个模态与阶段感知的模块化服务系统，支持各阶段解耦优化与自适应扩展。通过模态感知调度与自动伸缩，动态重构处理流程，在满足尾延迟SLO的同时显著降低资源开销。在128张GPU集群上使用真实生产轨迹测试，系统实现3.3-5.5倍吞吐提升，对应25%-41.3%的成本节约。
原文摘要 · Abstract (English)
Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environments poses significant challenges due to their complex architectures and heterogeneous characteristics across their multi-stage inference pipelines. We present the first comprehensive systems analysis of two prominent LMM architectures, decoder-only and cross-attention, across six representative open-source models, revealing key systems design implications. We also present an in-depth analysis of production LMM inference traces, uncovering unique workload characteristics, including variable, heavy-tailed request distributions and bursty traffic patterns. Based on these insights, we propose ModServe, a modular LMM serving system that decouples stages for independent optimization and adaptive scaling. ModServe dynamically reconfigures stages and handles bursty traffic with modality-aware scheduling and autoscaling to meet tail latency SLOs while minimizing costs. ModServe achieves 3.3-5.5x higher throughput (leading to 25-41.3% cost saving) while meeting SLOs on a 128-GPU cluster with production traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。