让预填充和解码同用一张显卡，兼顾低延迟与高吞吐。
RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
- 同一显卡并发执行预填充和解码，避免资源浪费。
- 在满足延迟SLO前提下，平均吞吐提升4.9倍，最高达32倍。
- 适配AMD显卡细粒度计算单元屏蔽，适合资源受限场景。
当前大语言模型推理服务普遍采用混合批处理和分离式部署两种技术。混合批处理将不同请求的预填充与解码令牌合并处理，虽提升资源利用率和吞吐量，但增加单字节延迟；分离式部署将计算密集型的预填充与带宽密集型的解码分离，优化服务等级目标（SLO），却导致资源利用不足及键值缓存传输开销。为克服上述局限，本文提出RAPID-Serve：一种在同一显卡上并行执行预填充与解码的技术，在满足延迟SLO的同时保持高吞吐与高效资源利用。此外，我们设计自适应资源管理机制，可选启用AMD Instinct™ GPU的细粒度计算单元屏蔽（CU masking）功能。实验表明，RAPID-Serve在无约束条件下平均吞吐提升1.7倍（最高达4.1倍），在满足SLO约束下平均提升4.9倍（最高超过32倍），相较现有最优方案表现优异，尤其适用于资源受限环境。
原文摘要 · Abstract (English)
Two widely adopted techniques for LLM inference serving systems today are hybrid batching and disaggregated serving. A hybrid batch combines prefill and decode tokens of different requests in the same batch to improve resource utilization and throughput at the cost of increased latency per token. In contrast, disaggregated serving decouples compute-bound prefill and bandwidth-bound decode phases to optimize for service level objectives (SLOs) at the cost of resource under-utilization and KV-cache transfer overheads. To address the limitations of these techniques, we propose RAPID-Serve: a technique to concurrently execute prefill and decode on the same GPU(s) to meet latency SLOs while maintaining high throughput and efficient resource utilization. Furthermore, we propose Adaptive Resource Management for runtime compute resource allocation, optionally leveraging CU masking (a fine-grained Compute Unit partitioning feature on AMD Instinct\textsuperscript{TM} GPUs). RAPID-Serve provides up to 4.1x (average 1.7x) unconstrained throughput improvement and 32x and higher (average 4.9x) throughput improvement under SLO constraints, showing it as an effective strategy compared to the state-of-the-art approaches, particularly in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。