拆解大模型推理的隐藏开销，帮开发者判断该优化软件还是硬件。
TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition
- 通过追踪分析将主机开销分解为框架、库和核启动三部分。
- 发现MoE模型每输出一个词多发8-11倍核，且单线程CPU性能影响10%-29%开销。
- 提出设备-主机平衡指数，指导优化方向：是改软件栈还是减计算量。
大语言模型推理广泛应用于交互式助手与智能体系统中。在延迟敏感场景下，推理时间常被主机侧开销主导。现有方法通常仅以聚合残差或启动/排队指标呈现此成本，难以定位需优化的执行层。本文提出TaxBreak，一种基于追踪的主机可见调度开销分解方法，将其分为框架转换时间、CUDA库转换时间和核启动路径时间三部分。我们在NVIDIA H100和H200系统上验证了TaxBreak，并据此提出主机-设备平衡指数（HDBI），该指数关联设备活跃执行与主机可见调度开销。在密集模型与混合专家（MoE）工作负载的预填充和解码阶段，我们发现聚合延迟、GPU空闲率或有界性比率单独使用会掩盖主要优化目标。TaxBreak则能区分应减少软件栈开销的情况与应减少设备端工作量的情况。进一步发现，MoE模型每输出一个词的核调用次数比密集模型多8-11倍；对于此类主机绑定负载，单线程CPU性能为一阶参数：更快的主机CPU可使调度开销降低10%-29%，端到端延迟最多提升14%，即使搭配时钟较慢的GPU。这些结果使TaxBreak成为评估优化投入应指向软件栈还是设备端执行的诊断工具。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latency-sensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, CPU single-thread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by 10-29% and improves end-to-end latency by up to 14%, even when paired with a slower-clocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。