为任意模态输入输出模型设计的分布式推理系统，提升吞吐与延迟表现。
Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models
- 通过灵活任务抽象实现模型组件解耦与独立扩展
- 支持多模态混合请求，吞吐最高提升3.81倍，尾延迟降低5.79倍
- 适用于需要动态处理多种模态组合的研究与工业场景
Any-to-Any模型是一类新兴的多模态模型，可接受文本、图像、视频、音频等任意组合输入，并生成对应输出。服务这类模型极具挑战：不同请求的输入输出模态组合导致计算路径各异，模型各组件的扩展特性也不同。本文提出Cornserve，一个面向通用Any-to-Any模型的分布式推理系统。Cornserve提供灵活的任务抽象以表达多模态计算图，支持组件解耦与独立扩展。其分布式运行时采用高效记录与重放执行模型，追踪数据依赖关系，并直接在生产者与消费者间传输张量数据。系统基于Kubernetes构建，包含约2.3万行Python代码，支持多样化的Any-to-Any模型，在测试中实现最高3.81倍吞吐提升与5.79倍尾延迟降低。Cornserve已开源，演示视频可在YouTube查看。
原文摘要 · Abstract (English)
Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81$\times$ higher throughput and 5.79$\times$ lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。