首个评测大模型生成低级内核的基准，加速新型芯片开发
KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware
- 用函数调用+反馈迭代的智能体流程生成内核
- 在3种新硬件上对20+任务测试，数步内生成正确内核
- 适合芯片设计、AI编译器研究者参考
新型人工智能加速器常采用创新指令集架构（ISA），需开发者手动编写底层内核，过程耗时且易出错，难以跨硬件扩展，延缓产品上市。尽管大语言模型在成熟GPU生态中已展现代码生成潜力，但其在新兴硬件新ISA上的表现尚不明确。我们提出KernelCraft：首个评估大模型智能体通过函数调用与反馈驱动工作流，为定制化加速器生成并优化低级内核的基准。在超过20个机器学习任务、每任务5种不同配置下，对4个主流推理模型进行评估。最强智能体可在数次迭代内为未见过的ISA生成功能正确的内核，并产出性能匹配甚至超越编译器基线的优化内核。结果表明KernelCraft有潜力显著加速加速器芯片研发周期。项目地址：https://kernelcraft-cam.github.io/
原文摘要 · Abstract (English)
New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels, a time-consuming and error-prone process that does not scale across hardware targets. This delays emerging hardware platforms from reaching the market. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark for evaluating an LLM agent's ability to generate and optimize low-level kernels for customized accelerators through a function-calling, feedback-driven workflow. We evaluate agent performance across three emerging accelerators on more than 20 machine-learning tasks, each with five diverse task configurations. Across four leading reasoning models, the strongest agents generate functionally correct kernels for unseen ISAs within a few refinement steps and produce optimized kernels that match or outperform compiler baselines. These results demonstrate KernelCraft's potential to accelerate the accelerator chip development cycle. KernelCraft is available at https://kernelcraft-cam.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。