arXiv:2607.11368cs.DCcs.LG2026-07被引 1

拆解量化加速的三大来源,发现运行时优化占主导。

Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

  • 用匹配的半精度中间栈分离运行时、核函数与量化影响。
  • 端到端加速达2.58倍,其中运行时贡献约三分之二。
  • 大模型更适合双卡配对,小模型则视负载选分片或多实例。

量化推理的加速效果常将权重格式、内核实现与运行时性能混为一谈。本文在四块NVIDIA RTX A5000 GPU(每块24 GiB,NVLink桥接成对)上进行归因分析,构建一个保持更快运行时但未使用量化内核的匹配中间栈,将整体加速分解为运行时部分与内核及量化部分。在贪婪解码下,完整栈实现2.58倍端到端加速,其中运行时改进贡献约三分之二(对数尺度)。在三个相似模型族中,内核与量化部分变化不超过1.5%。将单个实例跨四卡分片,性能提升远低于翻倍:性能剖析显示约80%的每标记损耗源于协调开销;同硬件下NVLink与PCIe实测带宽相近,排除链路带宽为瓶颈。是否采用分片实例或多个独立实例取决于工作负载与模型大小,大模型在所有场景下均偏好双卡配对实例。量化使可持续并发用户数在重现的半精度内存悬崖后扩展约四倍。采样模式与提示池差异被记录为有效性威胁。

原文摘要 · Abstract (English)

Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number. We present an attribution study on four NVIDIA RTX A5000 GPUs, 24 GiB each, on a single host with NVLink-bridged pairs. A matched intermediate stack that keeps the faster runtime without the quantized kernel splits the full speedup into a runtime part and a kernel and quantization part. Under matched greedy decoding the full stack reaches $2.58\times$ end to end, with the runtime change accounting for about two thirds of that gain on a logarithmic scale; across three similar model families the kernel and quantization part moves by at most 1.5%. Sharding one instance across all four cards falls well below doubling: a profiler trace attributes about 80% of the per token shortfall to coordination, and an NVLink versus PCIe control on the same hardware shows similar realized bandwidth on both links, pointing away from link bandwidth as the cause. Whether to run one sharded instance or several independent ones depends on the workload and the model, with the ranking reversing on the larger model: the smaller model splits between sharding and multiple instances by workload, while the larger model favors two paired instances on every workload. Quantization extends sustainable concurrent users roughly four times past a reproducible half precision memory cliff. Differences in sampling mode and prompt pool between the two stacks are documented as threats to validity.

量化加速运行时优化GPU并行性能归因

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。