提升CPU上大模型服务效率，解决多阶段资源冲突问题。
Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving
- 动态切换执行计划，消除预填充与解码阶段的资源干扰。
- 在5个CPU平台上平均提速2.01倍，最高延迟降低3.40倍。
- 自动适配硬件子结构，低开销生成高性能动态张量程序。
由于可用性、成本效益和边缘部署优势，CPU在大模型服务中至关重要。然而，在非分离部署约束下，预填充与解码阶段的资源需求存在冲突，现有方案无法避免跨阶段干扰，忽略次NUMA硬件结构，且动态形状内核性能不佳。我们提出Sandwich，一个全栈式CPU大模型服务系统，包含三项核心创新：(1) 无缝阶段间计划切换，消除跨阶段干扰；(2) TopoTree，一种基于树的硬件抽象，实现对子结构（如LLC切片）的感知式部分核心分配；(3) 快启动-精调的动态形状张量程序生成。在五个x86/ARM CPU平台上，Sandwich相比最先进系统平均实现2.01倍端到端加速，最高延迟降低3.40倍。其内核性能媲美静态编译器，但调优成本降低三个数量级。
原文摘要 · Abstract (English)
CPUs are critical for LLM serving due to their availability, cost efficiency, and edge applicability. However, efficient CPU serving is hindered by conflicting prefill/decode resource demands under non-disaggregated deployment constraints--existing solutions fail to avoid cross-phase interference, ignore sub-NUMA hardware structures, and deliver suboptimal dynamic-shape kernel performance. We propose Sandwich, a full-stack CPU LLM serving system with three core innovations addressing these challenges: (1) seamless phase-wise plan switching to eliminate cross-phase interference; (2) TopoTree, a tree-based hardware abstraction for automated substructure-aware (e.g., LLC slices) partial core allocation; (3) fast-start-then-finetune dynamic-shape tensor program generation. Across five x86/ARM CPU platforms, Sandwich achieves an average 2.01x end-to-end speedup and up to 3.40x latency reduction over state-of-the-art systems. Its kernels match static compiler performance with three orders of magnitude lower tuning cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。