arXiv:2606.12688cs.LGcs.AI2026-06

M*让多模态模型高效部署,支持任意组件组合与分布式优化。

M*: A Modular, Extensible, Serving System for Multimodal Models

论文配图:M*: A Modular, Extensible, Serving System for Multimodal Models
图 1 · 摘自论文原文
  • 将模型抽象为数据流图,按需遍历处理跨模态请求。
  • 在文本生成图像任务上比vLLM-Omni低20%端到端延迟,语音生成吞吐高2.7倍。
  • 适合需要灵活部署复杂多模态模型的研发团队使用。

我们正进入由视觉编码器、语言主干、扩散模型、音频编解码器、动作生成器和世界模型预测器等多样化组件构成的复合模型架构时代。这类架构支撑了统一多模态模型、全模态模型、语音-语言模型、视觉-语言-动作策略以及世界模型等多种应用。然而,现有模型服务框架基于对模型结构的狭隘假设,难以适应这种新架构多样性。本文提出M*,一个通用的复合模型服务系统。M*将模型表示为数据流图,将跨越多种模态和任务的请求作为图上的遍历过程处理。其核心思想是模块化抽象,支持任意组件组合、灵活部署于物理集群,并在分布式运行时中实现模型无关优化。该抽象称为Walk Graph,可简洁表达广泛模型家族。我们在代表性模型上实例化M*,发现在BAGEL数据集上,文本生成图像任务平均比vLLM-Omni降低20%端到端延迟;在Qwen3-Omni上,文本转语音任务实现实时因子降低2.9倍,吞吐提升2.7倍;在机器人规划任务上,相比V-JEPA 2-AC基线,性能最高提升12.5倍。本工作为高效部署复杂模型铺平道路,开发者投入极小。

原文摘要 · Abstract (English)

We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, action generators, and world-model predictors. Such architectures underpin a broad class of multimodal models, including unified multimodal models, omni models, speech-language models, vision-language-action policies, and world models. However, existing model serving frameworks were built on narrow assumptions about model structure, making them ill-suited to accommodate this new architectural diversity. Here we present M*, a universal serving system for efficient serving of composite AI models. M* represents models as dataflow graphs, processing requests spanning diverse modalities and tasks as traversals over these graphs. The core insight is a modular abstraction that supports arbitrary composition of model components, flexible placement onto a physical cluster, and model-agnostic optimizations within a distributed runtime. We call this abstraction the Walk Graph and show how it can concisely capture composite models from a broad range of families. We instantiate M* on representative models and find that it achieves, on average, 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL, while delivering up to 2.9x lower real-time factor and 2.7x higher throughput for text-to-speech workloads on Qwen3-Omni. M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x. Thus, our work paves the road towards more efficient serving of complex models with minimal developer effort.

多模态模型服务分布式高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。