解决大模型推理中不同LoRA适配器大小不一导致的性能偏差问题。
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
- 动态分配适配器到GPU，按秩大小智能调度。
- 实测吞吐量提升2倍，首字延迟降低9倍，少用50%显卡。
- 适合需要高效多租户部署大模型的工业场景。
低秩适配（LoRA）已成为大语言模型参数高效微调的主流方法，可快速适应多样领域。在生产环境中，基于LoRA的模型以大规模服务，形成数百个适配器共享一个基础模型的多租户环境。然而，现有先进服务系统在共批处理异构适配器时未考虑其秩（规模）差异，导致严重性能偏差，最终需增加更多GPU以满足服务级别目标（SLO）。现有优化聚焦于加载、缓存和核函数执行，忽略这种异质性，造成GPU资源利用率低下。我们提出LoRAServe，一种面向工作负载的动态适配器放置与路由框架，旨在缓解LoRA服务中的秩多样性问题。通过动态平衡适配器在GPU间的分布，并利用GPU Direct RDMA实现远程访问，LoRAServe在真实工作负载漂移下最大化吞吐量并最小化尾部延迟。在公司X的真实数据轨迹上评估显示，相比最先进系统，LoRAServe在满足SLO约束条件下，吞吐量最高提升2倍，首令牌延迟（TTFT）降低高达9倍，最多可减少50%的GPU使用量。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。