arXiv:2607.14541cs.AI2026-07被引 2

用真实生产数据测试大模型生成GPU内核,发现效果很差且常靠系统回退冒充成功。

Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

论文配图:Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
图 1 · 摘自论文原文
  • 基于真实集群推理日志构建30个算子440种形状的基准测试
  • 顶级模型在生产算子上仅达硬件性能上限的10%
  • 推出优化代理工具,可将失败的代码转为媲美人工调优的高效内核

现有GPU内核生成基准多来自合成或人工挑选的数据,与真实部署负载差异较大。我们提出Atrex-Bench,其30个算子和440种数据形状均直接从计算密集型、内存充裕的生产级GPU集群推理日志中采样。每个问题包含根据实际占用GPU时间加权的重要性分值,并按服务阶段分别计算,聚合得分聚焦于消耗最多服务时间的内核。在Atrex-Bench上评估六个前沿编码代理,即使最优的原生模型在生产算子上也仅达到约10%的硬件理论峰值性能;且仅凭正确性高估了能力,因为大量通过率来自PyTorch回退而非模型自动生成的内核。为缩小差距,我们同时发布Atrex-Kernel-Agent(AKA),一个基于性能剖析的内核优化代理,结合迭代测量-修正搜索、优化丢弃机制以突破停滞状态,以及包含298个参考内核文件和244份优化知识文档的分层知识库,并接入外部上游项目用于API/ISA查询。在受控案例研究中,该代理成功将零次飞速DSL回退转化为匹配或超越人工调优基准的真实高效内核。

原文摘要 · Abstract (English)

Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads. We present Atrex-Bench, a benchmark whose 30 operators and 440 shapes are sampled directly from full-cluster production inference traces of compute-limited, memory-rich GPUs. Each problem carries an importance weight derived from its share of observed GPU time, weighted by application card-hours and computed separately for the serving phases in which it runs, together with a per-problem roofline ceiling, so the aggregate score emphasizes the kernels that consume the most serving time. Evaluating six frontier coding agents on Atrex-Bench shows that even the best vanilla model reaches only ${\sim}10\%$ of the hardware roofline on production operators; and correctness alone overstates capability, since much of the apparent pass rate comes from PyTorch fallbacks rather than kernels the model wrote. To close this gap, we co-release Atrex-Kernel-Agent (AKA), a profile-driven kernel-optimization agent that combines iterative measure-revise search, optimization dropout for escaping stalled search contexts, and a layered GPU-optimization knowledge base (298 reference-kernel files and 244 optimization-knowledge documents, plus external upstream reference projects for API/ISA lookup). In a controlled case study, the agent converts zero-FlyDSL fallbacks into real kernels that match or exceed hand-tuned production baselines.

GPU内核大模型生成性能优化基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。