首个跨源跨芯片的LLM生成核函数基准，评测自动代码生成效能。
KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation
- 构建多源多芯片基准，评估LLM生成Triton核函数表现。
- 代理方法优于纯LLM采样，但跨平台性能下降超70%。
- 自主生成成本极高，每成功生成一个算子需数百万token。
大型语言模型（LLMs）显著提升了对高效加速器核函数的需求，但核函数开发仍高度专业化且耗时。近期LLM与智能体框架的兴起为自动核函数生成提供了新路径。然而，尽管进展迅速，目前仍缺乏全面基准来系统评估跨不同算子来源和异构硬件平台的LLM生成核函数性能。我们提出KernelGenBench，一个统一基准，用于系统评估基于LLM和智能体生成的Triton核函数，涵盖多样算子来源与异构硬件平台。其包含两个互补子基准：KernelGenBench-MS（多源），评估来自三个非标准PyTorch任务的210个算子；KernelGenBench-MC（多芯片），使用110个算子子集，在六种异构硬件平台上测量性能可移植性。大规模评估共消耗超过150亿token，结果显示：(1) 基于代理的方法持续优于纯LLM采样，而cuBLAS算子在所有方法中最具挑战性；(2) 生成性能在不同硬件平台间差异显著，即使近期专用核函数智能体也出现严重跨平台退化（如AutoKernel在NVIDIA上达87%，在Platform E降至25%）；(3) 自主核函数生成仍极昂贵，专用代理方法平均每成功生成一个算子需511万token（AKO4all达519万），远高于简单LLM采样方法。
原文摘要 · Abstract (English)
Large language models (LLMs) have significantly increased the demand for efficient accelerator kernels, but kernel development remains a highly specialized and labor-intensive task. The recent rise of LLMs and agentic frameworks offers a promising pathway toward automatic kernel generation. However, despite rapid progress, there is still no comprehensive benchmark to rigorously evaluate LLM-generated kernels across diverse operator sources or heterogeneous hardware platforms. We present KernelGenBench, a unified benchmark for systematically evaluating LLM- and agent-generated Triton kernels across diverse operator sources and heterogeneous hardware platforms. It comprises two complementary sub-benchmarks: KernelGenBench-MS (Multi-Source), evaluating 210 operators from three sources beyond standard PyTorch-centric tasks, and KernelGenBench-MC (Multi-Chip), measuring performance portability across six heterogeneous hardware platforms using a 110-operator subset. Our large-scale evaluation, consuming over 15 billion tokens, shows: (1) agent-based methods consistently outperform pure LLM sampling methods, while cuBLAS operators are the most challenging across all methods; (2) generation performance varies significantly across hardware platforms, with even recent kernel-specialized agents experiencing severe cross-platform degradation (e.g., AutoKernel drops from 87% on NVIDIA to 25% on Platform E); (3) autonomous kernel generation remains highly cost-intensive, with specialized agent methods averaging 5.11 million tokens per successful operator (AKO4all reaches 5.19 million), orders of magnitude higher than simple LLM sampling approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。