首次系统研究推理型大模型的部署表现,揭示其独特行为与优化难点。
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
- 对比推理型与通用大模型,发现内存波动、延迟请求等四类差异行为。
- 量化与推测解码可提升效率且小幅损失精度,缓存优化可能适得其反。
- 基于伽马分布的真实负载验证结果,支持核心发现,适合部署团队参考。
推理型大语言模型(RLLM)在数学、编程等复杂任务上表现优异,但其推理服务性能与行为尚未被充分探索,可能影响实际部署效果。本文开展全面的RLLM服务研究:首先通过初步实验对比RLLM与传统LLM的服务表现,揭示四类显著差异:(1)高内存占用与剧烈波动;(2)滞留请求(straggler requests);(3)自适应运行时间;(4)领域偏好。进一步评估现有推理优化技术对RLLM的有效性,发现模型量化与推测解码可提升系统效率并小幅牺牲精度,而前缀缓存与KV缓存量化反而可能降低精度或服务性能,尤其对小型RLLM。最后,在模拟伽马分布的真实工作负载下进行评估,跨多个数据集的结果均验证了前述结论。本研究为学术界与工业界提供关键洞察,助力推进RLLM推理服务发展。
原文摘要 · Abstract (English)
The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving performance and behavior of RLLM remains unexplored, which may undermine the deployment and utilization of RLLM in real-world scenario. To close this gap, in this paper, we conduct a comprehensive study of RLLM service. We first perform a pilot study on comparing the serving performance between RLLM and traditional LLM and reveal that there are several distinct differences regarding serving behavior: (1) significant memory usage and fluctuations; (2) straggler requests; (3) adaptive running time; (4) domain preference. Then we further investigate whether existing inference optimization techniques are valid for RLLM. Our main takeaways are that model quantization methods and speculative decoding can improve service system efficiency with small compromise to RLLM accuracy, while prefix caching, KV cache quantization may even degrade accuracy or serving performance for small RLLM. Lastly, we conduct evaluation under real world workload modeled by Gamma distribution to verify our findings. Empirical results of real world workload evaluation across different dataset are aligned with our main findings regarding RLLM serving. We hope our work can provide the research community and industry with insights to advance RLLM inference serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。