评测大模型生成的GPU内核，发现正确性与效率常不一致。
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
- 按任务类别系统评估176个内核生成任务的正确性与性能。
- 72%融合类任务失败，量化问题0成功，正确内核仍可能比基线慢。
- 需关注数值精度建模和硬件效率，而非仅语法正确。
基于大模型的Triton内核生成受到广泛关注，但一个根本性问题仍未解决：该能力在何处失效，原因何在？我们提出KernelBenchX，一个涵盖15个类别共176项任务的综合基准，用于分类评估正确性与硬件效率。对五种代表性方法的系统比较揭示三个核心发现：首先，任务结构对正确性的决定作用远超方法设计，类别解释了近三倍于方法的语义正确性方差（9.4% vs 3.3%），72%的Fusion任务在所有方法中均失败，而Math任务则一致通过；其次，迭代优化提升正确性但降低性能，编译成功率从52.3%升至68.8%，平均加速比从1.58×降至1.44×，新修复的内核始终低于初始正确内核（1.16× vs 1.58×）；第三，正确性不等于高效性，46.6%的正确内核比PyTorch eager基线更慢，跨硬件加速比差异达21.4×；此外，量化任务完全失败（0/30成功），暴露了对数值计算契约的深层误解，而非表面语法错误。这些发现表明，未来进展需聚焦全局协调、显式建模数值精度，并将硬件效率纳入生成过程。代码已开源。
原文摘要 · Abstract (English)
LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We present KernelBenchX, a benchmark designed to answer this question through category-aware evaluation of correctness and hardware efficiency across 176 tasks in 15 categories. Our systematic comparison of five representative methods yields three main findings. First, task structure determines correctness more than method design. Category explains nearly three times more variance in semantic correctness than method (9.4% vs 3.3% explained deviance), and 72% of Fusion tasks fail across all five methods while Math tasks are solved consistently. Second, iterative refinement improves correctness, but not performance. Across GEAK iterations, compile rate rises from 52.3% to 68.8% while average speedup declines from $1.58\times$ to $1.44\times$; newly rescued kernels consistently underperform persistently correct ones ($1.16\times$ vs $1.58\times$ speedup in round~0$\to$1). Third, correctness does not imply efficiency. 46.6% of correct kernels are slower than the PyTorch eager baseline, and cross-hardware speedup variance reaches $21.4\times$. Besides, quantization remains completely unsolved (0/30 successes) despite non-trivial compilation rates, revealing systematic misunderstanding of numerical computation contracts rather than surface-level syntax errors. These findings suggest that future progress depends on handling global coordination, explicitly modeling numerical precision, and incorporating hardware efficiency into generation. The code is available at https://github.com/BonnieW05/KernelBenchX
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。