发现浏览器端大模型推理的性能瓶颈是操作调度开销，而非内核质量。
Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference
- 提出序列化调度测量法，避免同步开销干扰结果
- 证实单次调度开销与数据类型无关，且在批大小为1时成主要瓶颈
- 建议优化方向为减少调度次数，适合浏览器部署场景
大型语言模型已部署于多种环境，包括浏览器和边缘设备，WebGPU作为现代跨平台标准被广泛采用。尽管基于浏览器的LLM推理引擎日益增多，但每次操作调度的开销仍缺乏准确评估。本文提出一种序列化调度测量方法，发现传统单操作测量会因调度与同步混杂而高估开销。通过该方法，我们测得调度开销与数据类型无关，并确认在批大小为1时，调度开销而非内核性能是主要瓶颈，且调度次数是根本原因。因此，结论为在批大小1下，优化浏览器端LLM推理的关键在于降低调度次数。研究指出，应在推理引擎和WebGPU规范中推进调度开销分摊，以实现实际可用的浏览器推理能力。
原文摘要 · Abstract (English)
Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet the overhead of WebGPU per-operation dispatch remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. Therefore, we conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。