提出统一抽象解决大模型推理中动态算子的调度难题。
Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
- 用事件张量统一描述分块任务依赖，支持形状与数据动态性。
- 编译器生成高性能持久化内核，大模型推理延迟达顶尖水平。
- 适合关注低延迟大模型服务与编译优化的研究者。
现代GPU工作负载，尤其是大语言模型（LLM）推理，受限于内核启动开销和粗粒度同步，导致内核间并行性不足。近期的巨内核技术将多个算子融合为单一持久内核，以消除启动间隙并暴露内核间并行性，但难以处理真实场景中的动态形状与数据依赖计算。本文提出事件张量（Event Tensor），一种用于动态巨内核的统一编译抽象。该抽象编码分块任务间的依赖关系，首次原生支持形状与数据依赖的动态性。基于此抽象，我们构建了事件张量编译器（ETC），通过静态与动态调度变换生成高性能持久内核。评估表明，ETC在保持领先大模型服务延迟的同时，显著降低系统预热开销。
原文摘要 · Abstract (English)
Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。