通过智能缓存管理，让多LoRA大模型推理快6成以上。
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
- 基于使用依赖关系统一管理LoRA和键值缓存
- 在显存空闲/繁忙时分别用不同策略交换缓存
- 显著降低首次输出延迟，适合高并发推理场景
多低秩适配器（Multi-LoRA）在任务特定的大语言模型应用中日益流行。为提升多LoRA服务性能，将热键值（KV）缓存和LoRA适配器缓存在加速器的高带宽内存中可有效提升推理效率。然而，现有多LoRA推理系统未能优化服务性能指标如首词延迟（TTFT），且在缓存LoRA与KV时忽略了使用依赖关系。为此，我们提出FASTLIBRA，一种多LoRA缓存系统，以优化服务性能。FASTLIBRA包含一个依赖感知的缓存管理器和一个性能驱动的缓存交换器。缓存管理器通过统一缓存池维护推理过程中LoRA与KV缓存间的使用依赖关系。缓存交换器根据统一成本模型，在高带宽内存（HBM）空闲或繁忙时，分别决定LoRA和KV缓存的加载或替换。实验结果表明，相比当前最优方案，该方法平均降低首词延迟63.4%。
原文摘要 · Abstract (English)
Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。