通过预取权重和缓存，显著降低大模型推理通信开销。
PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving
- 在通信时将模型权重和KV缓存预取到芯片缓存中
- 实测端到端速度最高提升1.6倍，成本性能比提升1.25倍
- 适合大规模分布式大模型部署场景
大型语言模型通常在由大量GPU/NPU组成的集群上进行服务。然而，设备间的通信带来显著开销，增加了推理延迟和成本，并限制了可扩展性。以往工作尝试通过重叠通信与计算来缓解该问题，但受操作间数据依赖性的严重制约。本文提出PRESERVE框架，在通信操作期间将模型权重和KV缓存从片外HBM内存预取至AI加速器的片上缓存，相比先前方法具有多项优势与性能提升。在商用AI加速器上的广泛实验表明，对主流开源大模型的端到端推理速度最高提升1.6倍。此外，通过设计空间探索，我们识别出最优硬件配置，选择最佳L2缓存大小后，性能/成本比进一步提升1.25倍。结果表明，PRESERVE具备缓解内存瓶颈与通信开销的潜力，为提升大模型推理系统的性能与可扩展性提供有效方案。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically served from clusters of GPUs/NPUs that consist of large number of devices. Unfortunately, communication between these devices incurs significant overhead, increasing the inference latency and cost while limiting the scalability. Prior work addressed this issue by overlapping communication with compute, but has severe limitations due to the data dependencies between these operations. In this paper, we propose PRESERVE, a novel framework that prefetches model weights and KV-cache from off-chip HBM memory to the on-chip cache of AI accelerators during the communication operations, which offers various advantages and performance improvements compared to prior methods. Through extensive experiments conducted on commercial AI accelerators, we demonstrate up to 1.6x end-to-end speedup on state-of-the-art, open-source LLMs. Additionally, we perform a design space exploration that identifies the optimal hardware configuration for the proposed method, showing a further 1.25x improvement in performance per cost by selecting the optimal L2 cache size. Our results show that PRESERVE has the potential to mitigate the memory bottlenecks and communication overheads, offering a solution to improve the performance and scalability of the LLM inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。