发现并解决大模型推理评测中的系统性偏差问题。
Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
- 用多进程框架替代单进程评测，消除客户端排队瓶颈。
- 实测显示高并发下原方法使响应延迟夸大2.3倍以上。
- 适合关注生产级大模型性能评估的工程师和研究者。
随着大语言模型从研究环境进入生产部署，满足严格的服务水平目标（SLO）性能评估变得至关重要。然而，当前评估方法在大规模下存在严重测量偏差。我们证明，广泛使用的基准测试工具依赖单进程、异步驱动架构，在高并发时引入根本性的客户端队列瓶颈。通过将评测客户端建模为M/G/1队列，数学上展示了Python全局解释器锁（GIL）会随请求速率增加，人为放大首次令牌时间（TTFT）和每输出令牌时间（TPOT）指标。为解决这一系统性误差，我们提出一种无偏的多进程评估框架，有效分散客户端负载，实现可忽略的队列开销。此外，我们形式化定义了归一化每输出令牌时间（NTPOT）复合指标，以稳健地分摊端到端延迟，包括预填充和调度延迟，覆盖不同序列长度。实证评估表明，该方法能准确隔离纯服务引擎性能，实现超千次每秒查询量下的可复现、精准的大模型性能分析。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) transition from research environments to production deployments, evaluating their performance against strict Service Level Objectives (SLOs) has become critical. However, current evaluation methodologies suffer from severe measurement bias at scale. We demonstrate that widely used benchmarking utilities rely on single-process, asyncio-driven architectures that introduce fundamental client-side queuing bottlenecks under high concurrency. By modeling the benchmarking client as an $M/G/1$ queue, we mathematically demonstrate how the Python Global Interpreter Lock (GIL) artificially inflates Time to First Token (TTFT) and Time Per Output Token (TPOT) metrics as request rates scale. To resolve this systematic inaccuracy, we propose an unbiased, multi-process evaluation framework that effectively distributes client-side load, ensuring negligible queuing overhead. Furthermore, we formalize a composite metric, Normalized Time Per Output Token (NTPOT), to robustly amortize end-to-end latency, including prefill and scheduling delays across sequence lengths. Our empirical evaluation demonstrates that this methodology successfully isolates pure serving engine performance, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。