揭露大模型推理系统评估中的三大常见陷阱,推动更真实可靠的性能测试。
On Evaluating Performance of LLM Inference Serving Systems
- 提出评估反模式清单,覆盖基准公平性、设置与指标设计三方面。
- 发现现有方法常因工作负载不真实或指标掩盖波动而得出错误结论。
- 适合研究者和工程师提升评测严谨性,避免误导性结果。
大语言模型(LLM)推理系统的快速发展带来了显著的效率提升,但我们的系统性分析揭示,当前评估方法普遍存在根本性缺陷,常表现为评估反模式,掩盖真实性能并阻碍科学进步。通过对近期系统的全面考察,我们识别出三个关键维度的重复反模式:基准公平性、评估设置和指标设计。这些反模式对LLM推理尤为严重,因其具有预填充与解码双阶段特性、处理高度异构的工作负载,并对交互式使用有严格的时序要求。我们展示常见反模式如何导致误导性结论,例如:将工程努力与算法创新混淆的基线比较、无法代表生产场景的工作负载选择、以及隐藏生成阻塞等性能波动的指标归一化。为此,我们基于分析提供一份综合检查清单,建立识别与规避反模式的框架,以实现稳健的LLM推理评估。为验证其应用,我们以推测解码为例进行案例研究,该技术的突发性、非均匀标记生成极易在典型反模式评估下被误读。本工作为评估方法奠定严谨基础,使比较有意义、结果可复现,最终推动真正进展,让评估贴近真实需求。
原文摘要 · Abstract (English)
The rapid evolution of Large Language Model (LLM) inference systems has yielded significant efficiency improvements. However, our systematic analysis reveals that current evaluation methodologies frequently exhibit fundamental flaws, often manifesting as common evaluation anti-patterns that obscure true performance characteristics and impede scientific progress. Through a comprehensive examination of recent systems, we identify recurring anti-patterns across three key dimensions: Baseline Fairness, Evaluation Setup, and Metric Design. These anti-patterns are uniquely problematic for LLM inference due to its dual-phase nature combining distinct prefill and decode operations, its handling of highly heterogeneous workloads, and its strict temporal requirements for interactive use. We demonstrate how common anti-patterns -- such as inadequate baseline comparisons that conflate engineering effort with algorithmic novelty, workload selections that fail to represent production scenarios, and metric normalizations that hide substantial performance variability like generation stalls-lead to misleading conclusions. To address these challenges, we provide a comprehensive checklist derived from our analysis, establishing a framework for recognizing and avoiding these anti-patterns in favor of robust LLM inference evaluation. To demonstrate the practical application of our framework, we present a case study analyzing speculative decoding, a technique whose bursty, non-uniform token generation is easily misinterpreted when evaluated using approaches characteristic of these anti-patterns. Our work establishes a rigorous foundation for evaluation methodology, enabling meaningful comparisons, ensuring reproducible results, and ultimately accelerating genuine progress in LLM inference systems by moving beyond common anti-patterns to align evaluation with real-world requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。