Vortex让AI服务在低延迟和高吞吐下稳定运行，适合对响应速度要求高的场景。
Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements
- 采用SLO优先架构，避免传统批处理带来的延迟波动。
- 在多种负载下延迟更低且更稳定，相同条件下请求速率超现有平台两倍以上。
- 支持RDMA时优势更明显，适合高要求的AI代理与实时交互应用。
越来越多的ML推理与知识检索服务需要同时应对终端用户的交互查询以及集成在端用户应用中的AI代理所产生的高负载请求流。我们核心假设是，后者将带来严格的服务级别延迟目标（SLO）。现有ML服务系统通过批处理优化吞吐量，但会引入不可预测的尾部延迟。Vortex实现了SLO优先的设计理念。对于相同任务，Vortex在各种工作负载下均显著低于TorchServe和Ray Serve的延迟，且更为稳定，通常可在超过两倍的请求速率下达成既定的SLO目标。当支持RDMA时，Vortex的优势更加突出。
原文摘要 · Abstract (English)
There is growing interest in deploying ML inference and knowledge retrieval as services that could support both interactive queries by end users and more demanding request flows that arise from AIs integrated into a end-user applications and deployed as agents. Our central premise is that these latter cases will bring service level latency objectives (SLOs). Existing ML serving platforms use batching to optimize for high throughput, exposing them to unpredictable tail latencies. Vortex enables an SLO-first approach. For identical tasks, Vortex's pipelines achieve significantly lower and more stable latencies than TorchServe and Ray Serve over a wide range of workloads, often enabling a given SLO target at more than twice the request rate. When RDMA is available, the Vortex advantage is even more significant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。