物理AI推理中，内存速度越快，延迟改善越有限，因启动开销成瓶颈。
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
- 发现物理AI单流推理受启动开销制约，非单纯内存带宽问题。
- 在H100上启用CUDA Graph可降低延迟25.9%，而L4仅降2.8%。
- 量化虽减少内存流量，但实际加速效果远低于预期，需优化运行时效率。
物理AI系统（如机器人、自动驾驶、边缘协作者）常面临与云大模型服务不同的推理负载：单流、批大小为1的自回归解码，即一个机器人、摄像头或用户会话等待下一个令牌。该负载通常被描述为内存带宽受限。每个解码步骤需流式传输模型权重和活跃的键值缓存，因此延迟应随峰值显存带宽变化。我们发现这一说法正确但不完整。我们在四款NVIDIA GPU（H100 SXM5、A100-80GB SXM4、L40S、L4）上测量了三款7–8B级GQA变压器在2048至16384上下文长度下的批量1解码，共获得44个有效测试点，在受控的bf16 SDPA设置下进行。随着峰值带宽提升，实际达到的显存带宽占比反而下降。以Qwen-2.5-7B、ctx=2048为例，L4达到其理论内存下限的约81%，而H100仅达27%。物理AI解码虽由内存主导，但更快的内存并未带来成比例的延迟降低。通过CUDA Graph A/B实验验证，发现在H100上（ctx=2048），启用CUDA Graph使10次全新会话的解码延迟降低1.259倍，95%置信区间为1.253至1.267；而在L4上仅降低1.028倍。这揭示了一个在快速GPU上显现、在慢速带宽受限GPU上隐藏的启动侧开销。部署启示是：内存节省只有在运行时真正实现才有效。在L4上，bf16解码已接近内存下限，但常见量化路径未能恢复预期的4倍权重流量减少：bnb-nf4达59.36毫秒/步，AutoAWQ+Marlin达45.24毫秒/步（基线62.32毫秒），而使用Ada调优int4内核的GPTQ+ExLlamaV2可达17.36毫秒/步。
原文摘要 · Abstract (English)
Physical AI systems, including robots, autonomous vehicles, embodied agents and edge copilots, often run a different inference workload from cloud LLM serving: single-stream, batch-1 autoregressive decode, where one robot, camera feed or user session waits on the next token. This workload is usually described as memory-bandwidth-bound. Each decode step streams model weights and the active KV cache, so latency should scale with peak HBM bandwidth. We show that this account is true but incomplete. We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs: H100 SXM5, A100-80GB SXM4, L40S and L4. We evaluate context lengths from 2048 to 16384, producing 44 valid cells under a controlled bf16 SDPA setup. The achieved fraction of peak HBM bandwidth falls as peak bandwidth rises. On the headline Qwen-2.5-7B ctx=2048 cell, an L4 reaches roughly 81 percent of its analytic memory floor, while an H100 reaches only 27 percent. Physical-AI decode is memory-dominated, but faster memory does not translate into proportional latency gains. We test the missing term with a CUDA Graphs A/B experiment. On H100 at ctx=2048, CUDA Graphs improves decode latency by 1.259x across N=10 fresh sessions, with a 95 percent bootstrap confidence interval of 1.253 to 1.267. On L4, the same intervention gives only 1.028x. This isolates a launch-side overhead that becomes visible on fast GPUs but remains mostly hidden on slower, bandwidth-bound GPUs. The deployment implication is that memory savings matter only when the runtime realises them. On L4, bf16 decode sits close to the memory floor, but common quantised paths do not recover the expected 4x weight-traffic reduction: bnb-nf4 reaches 59.36 ms/step and AutoAWQ+Marlin reaches 45.24 ms/step from a 62.32 ms bf16 baseline. GPTQ+ExLlamaV2, with Ada-tuned int4 kernels, reaches 17.36 ms/step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。