混合编译与图执行，显著降低大模型推理延迟。
Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference
- 动态部分用JIT编译，静态部分用CUDA图复用，减少启动开销。
- 在10-500词输入下，首令牌耗时降低66.0%，尾部延迟更低。
- 适合对延迟敏感的实时交互场景，如对话系统部署。
大语言模型在自然语言和多模态任务中表现优异，但其实际部署受限于推理延迟和内核启动开销，尤其在交互式短序列场景中更为突出。本文提出一种混合运行时框架，结合即时（JIT）编译与CUDA图执行，在保持自回归解码灵活性的同时降低启动开销。该框架将Transformer推理拆分为静态部分（通过CUDA图重放执行）和动态部分（由JIT编译内核处理），支持跨解码步骤的异步图捕获与复用。我们在单卡、批大小为1的条件下，对LLaMA-2 7B在10至500词提示长度上进行了评估。实验结果表明，该方法可将首令牌时间（TTFT）降低最多66.0%，且相比TensorRT-LLM在该场景下具有更低的P99延迟。结果表明，混合JIT-CUDA图执行能有效降低短序列大模型推理的延迟与波动，是低延迟AI应用的实用优化策略。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains constrained by inference latency and kernel launch overhead, particularly in interactive, short-sequence settings. This paper presents a hybrid runtime framework that combines Just-In-Time (JIT) compilation with CUDA Graph execution to reduce launch overhead while preserving runtime flexibility during autoregressive decoding. The framework partitions transformer inference into static components executed via CUDA Graph replay and dynamic components handled through JIT-compiled kernels, enabling asynchronous graph capture and reuse across decoding steps. We evaluate the proposed approach on LLaMA-2 7B using single-GPU, batch-size-one inference across prompt lengths from 10 to 500 tokens. Experimental results show that the hybrid runtime reduces Time-to-First-Token (TTFT) by up to 66.0% and achieves lower P99 latency compared with TensorRT-LLM in this regime. These results indicate that hybrid JIT-CUDA Graph execution can effectively reduce inference latency and variance for short-sequence LLM workloads, making it a practical optimization strategy for latency-sensitive AI applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。