LLMServingSim2.0统一模拟异构硬件与服务技术，助力大模型部署优化。
LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure
- 采用追踪驱动建模与操作级延迟分析，一键集成新加速器。
- 在TPU上实现18.5倍代码量减少，GPU模拟误差仅1.9%。
- 适合硬件开发者和大模型服务方快速验证系统设计。
本文提出LLMServingSim2.0，一个面向大规模大模型服务系统的统一仿真平台，用于探索异构硬件与服务技术。针对前代工具在硬件建模抽象不清、支持服务技术有限的问题，该平台引入追踪驱动性能建模与操作级延迟分析仪，支持通过单条命令集成新加速器。同时，嵌入最新服务策略，开放请求路由、缓存管理与调度策略接口。在TPU案例中，其分析仪代码量仅为前代的1/18.5，且保持低误差；实验表明其对基于GPU的服务系统仿真误差仅为1.9%，同时具备可接受的仿真效率，是软硬件协同优化的理想平台。
原文摘要 · Abstract (English)
This paper introduces LLMServingSim2.0, a system simulator designed for exploring heterogeneous hardware in large-scale LLM serving systems. LLMServingSim2.0 addresses two key limitations of its predecessor: (1) integrating hardware models into system-level simulators is non-trivial due to the lack of a clear abstraction, and (2) existing simulators support only a narrow subset of serving techniques, leaving no infrastructure that captures the breadth of approaches in modern LLM serving. To overcome these issues, LLMServingSim2.0 adopts trace-driven performance modeling, accompanied by an operator-level latency profiler, enabling the integration of new accelerators with a single command. It further embeds up-to-date serving techniques while exposing flexible interfaces for request routing, cache management, and scheduling policies. In a TPU case study, our profiler requires 18.5x fewer LoC and outperforms the predecessor's hardware-simulator integration, demonstrating LLMServingSim2.0's low-effort hardware extensibility. Our experiments further show that LLMServingSim2.0 reproduces GPU-based LLM serving with 1.9% error, while maintaining practical simulation time, making it a comprehensive platform for both hardware developers and LLM service providers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。