HeraSys通过细粒度协同调度，显著降低大模型工作流的延迟并提升吞吐量。
HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
- 通过节点合并与复用，消除多工作流间的计算冗余。
- 在严格延迟约束下，尾部延迟降低2.17倍，吞吐量提升1.85倍。
- 适合高并发、多租户的复杂大模型服务场景。
大型语言模型（LLMs）的普及使得服务系统从处理单个请求转向协调高并发、多租户的智能体工作流。然而，现有方案通常只关注工作流内部优化，忽视了跨工作流优化的巨大潜力。本文提出HeraSys，一种面向并发工作流端到端性能优化的大模型服务系统。通过细粒度编排，HeraSys利用结构化节点合并与复用，消除跨工作流计算冗余。此外，引入负载感知联合调度策略，动态评估跨查询与内查询优先级以调整执行顺序。结合资源倾斜机制、自适应批处理与流水线分解，HeraSys有效缓解尾部延迟，同时保持低平均延迟，显著提升系统吞吐量。大量实验表明，在严格延迟保障下，HeraSys将P99延迟降低最高达2.17倍，吞吐量提升最高达1.85倍。
原文摘要 · Abstract (English)
The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration, HeraSys eliminates cross-workflow computational redundancy via structural node merging and reuse. Furthermore, HeraSys introduces a load-aware joint scheduling policy that dynamically manages execution order by evaluating both inter- and intra-query priorities. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2.17$\times$ and increases serving throughput by up to 1.85$\times$ under strict latency guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。