用概率模型预测大模型请求，显著降低响应延迟。
Efficient Serving of LLM Applications with Probabilistic Demand Modeling
- 构建概率需求图建模大模型请求动态
- 调度优化使平均完成时间降低70%以上
- 适合高并发大模型服务场景的系统设计
基于大语言模型（LLM）的应用在多样后端上具有动态变化的资源需求。现有服务系统将资源需求视为黑盒，因队列顺序不当和后端预热延迟，影响整体效率。本文发现，通过概率需求图（PDGraph）可通用且准确建模此类需求。据此提出Hermes系统，利用PDGraph实现高效服务：采用吉廷斯策略确定调度顺序以最小化平均任务完成时间，并根据模型预判时机提前预热冷后端。在多种真实大模型应用上的实验表明，Hermes能有效提升服务效率，平均完成时间减少超70%，P95完成时间降低超80%。
原文摘要 · Abstract (English)
Applications based on Large Language Models (LLMs) contains a series of tasks to address real-world problems with boosted capability, which have dynamic demand volumes on diverse backends. Existing serving systems treat the resource demands of LLM applications as a blackbox, compromising end-to-end efficiency due to improper queuing order and backend warm up latency. We find that the resource demands of LLM applications can be modeled in a general and accurate manner with Probabilistic Demand Graph (PDGraph). We then propose Hermes, which leverages PDGraph for efficient serving of LLM applications. Confronting probabilistic demand description, Hermes applies the Gittins policy to determine the scheduling order that can minimize the average application completion time. It also uses the PDGraph model to help prewarm cold backends at proper moments. Experiments with diverse LLM applications confirm that Hermes can effectively improve the application serving efficiency, reducing the average completion time by over 70% and the P95 completion time by over 80%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。