Harli通过共置微调任务提升LLM推理的显存利用率。
Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
- 将计算密集型的PEFT微调与解码任务共置，缓解显存瓶颈。
- 平均提升微调吞吐量46.2%（最高达92.0%）。
- 适合需要高资源利用率的模型即服务场景。
大型语言模型（LLMs）正越来越多地部署在模型即服务（MaaS）平台中。为满足严格的服务质量（QoS）要求，现有推理系统将预填充和解码阶段分离。然而，在动态负载下，解码实例常因内存受限和批处理不足导致GPU利用率低下，造成计算资源浪费。本文提出Harli，一种通过将参数高效微调（PEFT）任务与LLM解码实例共置来提升GPU利用率的推理系统。由于PEFT任务具有计算密集、内存效率高的特点，适合作为安全共置目标。Harli通过三个核心组件解决关键挑战：统一内存分配器实现运行时内存复用，两阶段延迟预测器建模解码延迟，以及基于QoS保障的吞吐最大化调度器以最大化吞吐。实验表明，相较于最先进的服务系统，Harli在微调吞吐上平均提升46.2%（最高达92.0%），同时严格保证推理解码的QoS要求。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed under the Model-as-a-Service (MaaS) paradigm. To meet stringent quality-of-service (QoS) requirements, existing LLM serving systems disaggregate the prefill and decode phases of inference. However, decode instances often experience low GPU utilization due to their memory-bound nature and insufficient batching in dynamic workloads, leaving compute resources underutilized. We introduce Harli, a serving system that improves GPU utilization by co-locating parameter-efficient finetuning (PEFT) tasks with LLM decode instances. PEFT tasks are compute-bound and memory-efficient, making them ideal candidates for safe co-location. Specifically, Harli addresses key challenges--limited memory and unpredictable interference--using three components: a unified memory allocator for runtime memory reuse, a two-stage latency predictor for decode latency modeling, and a QoS-guaranteed throughput-maximizing scheduler for throughput maximization. Experimental results show that Harli improves the finetune throughput by 46.2% on average (up to 92.0%) over state-of-the-art serving systems, while maintaining strict QoS guarantees for inference decode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。