arXiv:2605.16819cs.CLcs.AI2026-05

评测AI代理优化GPU内核的泛化能力,发现生成代码易依赖特定数据形状。

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

论文配图:AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents
图 1 · 摘自论文原文
  • 构建全流程代理评测框架,包含编译、正确性与性能验证
  • 实测最高提速6.89倍,但跨配置泛化时正确率显著下降
  • 适合研究智能编程代理与高性能计算的开发者使用

GPU内核优化对高效深度学习系统至关重要,但编写高性能内核仍需大量底层经验。近期的AI编程代理可迭代读取代码、调用编译器与性能分析工具并优化实现,但现有内核基准测试仅评估单次LLM调用,未涵盖完整代理工作流,也缺乏内核间优化及未见配置泛化测试。我们提出AgentKernelArena,一个开源基准,用于评估AI编程代理在GPU内核优化中的表现。该基准包含196个任务,涵盖HIP-to-HIP、Triton-to-Triton优化及PyTorch-to-HIP转换,通过隔离环境中的门控编译、正确性与性能检查、集中评分和未见配置泛化协议,全面评估代理工作流。在包括Cursor Agent、Claude Code和Codex Agent在内的生产级代理中,多数任务类别编译成功率接近100%,正确率高,最优配置下在PyTorch-to-HIP任务中平均提速达6.89倍,HIP-to-HIP为6.69倍,Triton-to-Triton为2.13倍。未见配置测试显示,HIP-to-HIP与Triton-to-Triton优化大多可泛化至未见过的数据形状,而PyTorch-to-HIP任务则出现显著正确率下降,表明代理从头生成代码时常隐含特定形状假设。AgentKernelArena是一个模块化、可扩展的框架,支持跨代理、任务与硬件目标的严谨评估。

原文摘要 · Abstract (English)

GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial low-level expertise. Recent AI coding agents can iteratively read code, invoke compilers and profilers, and refine implementations, yet existing kernel benchmarks evaluate single LLM calls rather than full agent workflows, and none include both kernel-to-kernel optimization and unseen-configuration generalization testing. We present AgentKernelArena, an open-source benchmark for measuring AI coding agents on GPU kernel optimization. The benchmark contains 196 tasks spanning HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation, and evaluates complete agent workflows in isolated workspaces using gated compilation, correctness, and performance checks, centralized scoring and an unseen-configuration generalization protocol that tests whether optimizations transfer to input configurations the agent never observed. Across production agents including Cursor Agent, Claude Code, and Codex Agent, we find near-perfect compilation and high correctness rates on most task categories, with the strongest configurations achieving mean speedups of up to 6.89x on PyTorch-to-HIP, 6.69x on HIP-to-HIP, and 2.13x on Triton-to-Triton tasks. Our unseen-configuration evaluation shows that HIP-to-HIP and Triton-to-Triton optimizations largely transfer to unseen input shapes, while PyTorch-to-HIP exhibits substantial correctness drops, indicating that agents generating kernels from scratch frequently hardcode shape-specific assumptions. AgentKernelArena is designed as a modular, extensible framework for rigorous evaluation of agentic GPU kernel optimization across agents, tasks, and hardware targets.

AI编程GPU优化代理评测泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。