动态调整模型分片策略，提升大模型推理吞吐量。
Seesaw: High-throughput LLM Inference via Model Re-sharding
- 根据预填充和解码阶段特点，动态切换并行策略
- 相比vLLM平均提速1.36倍，最高达1.78倍
- 适合追求高吞吐的生产级大模型部署场景
为提升分布式大语言模型推理效率，现有方法采用张量并行、流水线并行等策略。然而，预填充与解码两个阶段的计算特性差异，使得单一静态并行策略难以同时优化两阶段性能。本文提出Seesaw推理引擎，通过动态模型重分片技术，在不同阶段灵活切换并行策略，最大化两阶段吞吐。为降低重分片开销并提升计算效率，引入分层KV缓存缓冲与最小化切换调度策略，协同减少频繁阶段切换带来的开销，同时保证最大批处理效率。实验表明，Seesaw相较最广泛使用的vLLM推理引擎，吞吐量最高提升1.78倍（平均1.36倍）。
原文摘要 · Abstract (English)
To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed. However, the distinct computational characteristics inherent in the two stages of LLM inference-prefilling and decoding-render a single static parallelization strategy insufficient for the effective optimization of both stages. In this work, we present Seesaw, an LLM inference engine optimized for throughput-oriented tasks. The key idea behind Seesaw is dynamic model re-sharding, a technique that facilitates the dynamic reconfiguration of parallelization strategies across stages, thereby maximizing throughput at both phases. To mitigate re-sharding overhead and optimize computational efficiency, we employ tiered KV cache buffering and transition-minimizing scheduling. These approaches work synergistically to reduce the overhead caused by frequent stage transitions while ensuring maximum batching efficiency. Our evaluation demonstrates that Seesaw achieves a throughput increase of up to 1.78x (1.36x on average) compared to vLLM, the most widely used state-of-the-art LLM inference engine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。