Coral通过智能调度多模型在异构GPU上的部署，显著降低推理成本。
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
- 动态优化多模型在不同GPU上的资源分配与服务策略
- 在资源紧张时提升2.39倍有效吞吐量，成本降低最多2.79倍
- 适合需要低成本、高弹性部署多个大模型的云服务场景
大型语言模型（LLMs）的应用日益分散，无单一模型占据主导。同时，云服务商提供大量中端及旧款GPU，其单位成本性能优于顶级硬件且更易获取。为高效利用这些异构资源并行服务多个LLM，我们提出Coral——一种自适应的异构感知多模型推理系统。其核心思想是联合优化所有模型的资源分配与副本服务策略。为应对吞吐量需求和资源可用性的动态变化，Coral采用无损两阶段分解方法，在将在线求解时间从小时级缩短至数十秒的同时保持全局最优性。在6个模型和20种GPU配置上的评估表明，Coral相比最佳基线可降低2.79倍服务成本，并在资源稀缺条件下实现最高2.39倍的有效吞吐量提升。
原文摘要 · Abstract (English)
The usage of large language models (LLMs) has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79$\times$ over the best baseline, and delivers up to 2.39$\times$ higher goodput under scarce resource availability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。