用数据驱动方法优化大规模LLM适配器部署,大幅减少所需GPU数量。
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
- 构建数字孪生系统+轻量机器学习模型,精准预测每块GPU的最高吞吐能力。
- 在不丢请求、不报内存错误前提下,平均降低60%的GPU使用量。
- 适合需要高效部署海量LLM适配器的云服务或AI平台团队。
大型语言模型(LLM)适配器可实现低成本模型定制化,但在分布式服务中需同时托管数百个适配器,带来复杂的缓存与调度挑战。现有工作多关注延迟和吞吐优化,而通过接近峰值利用率来最小化GPU资源的方案仍研究不足。本文提出一种数据驱动流水线:针对给定工作负载,计算出以最少GPU数量支撑该负载的适配器部署方案,同时避免请求饥饿和显存错误。方法基于真实服务行为学习的准确性能预测,识别每块GPU的最大可行吞吐量。流水线包含三个组件:(i) 针对LLM适配器服务设计的数字孪生(DT),(ii) 在DT生成数据上训练的轻量化机器学习模型,(iii) 利用ML性能估计进行贪心放置的算法。DT以高保真度模拟真实系统动态,吞吐预测误差低于5%,执行速度比完整LLM基准测试快达90倍,覆盖可预测与不可预测工作负载。所学的机器学习模型进一步加速性能估计,精度损失极小,支持可扩展优化。实验表明,该流水线显著提升GPU效率,在评估场景中平均将维持目标工作负载所需的GPU数量减少60%。此外,该框架还可扩展至延迟最小化等其他目标,适用于未来大规模LLM服务基础设施。
原文摘要 · Abstract (English)
Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently. While prior work has largely focused on latency and throughput optimization, minimizing GPU resource requirements through near-peak utilization remains largely underexplored. This paper presents a data-driven pipeline that, for a given workload, computes an adapter placement that serves the workload with the minimum number of GPUs while avoiding request starvation and GPU memory errors. To that end, the approach identifies the maximum feasible throughput attainable on each GPU by leveraging accurate performance predictions learned from real serving behavior. The proposed pipeline integrates three components: (i) a Digital Twin (DT) tailored to LLM-adapter serving, (ii) a distilled machine learning (ML) model trained on DT-generated data, and (iii) a greedy placement algorithm that exploits ML-based performance estimates to maximize GPU efficiency. The DT emulates real system dynamics with high fidelity, achieving below 5% throughput estimation error while executing up to 90x faster than full LLM benchmarking across both predictable and unpredictable workloads. The learned ML models further accelerate performance estimation with marginal accuracy degradation, enabling scalable optimization. Experimental results demonstrate that the pipeline substantially improves GPU efficiency, reducing the number of GPUs required to sustain target workloads by 60\% on average across the evaluated scenarios. Beyond GPU efficiency, the pipeline can be adapted to alternative objectives, such as latency minimization, highlighting its versatility for future large-scale LLM serving infrastructures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。