首个支持多平台的深度学习核生成评测基准,提升LLM生成效率。
MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation
- 构建跨平台核生成评测框架,覆盖14类285个任务
- 7个主流LLM测试显示平台泛化能力差异显著
- 提供类别感知提示策略,提升生成质量
利用大语言模型(LLMs)自动生成深度学习(DL)核,可显著降低高性能算子实现所需的人工投入和硬件专知。然而,现有评测基准在硬件支持、核分类粒度和任务覆盖均衡性方面存在局限。为此,我们提出MultiKernelBench,首个面向基于LLM的DL核生成的综合性多平台评测基准。该基准涵盖14个明确定义的核类别,共285个任务,支持Nvidia GPU、华为NPU和Google TPU三大硬件平台。为保障可扩展性,设计模块化后端抽象层,将平台相关逻辑与核心评测架构解耦,便于未来集成新硬件。进一步提出一种简单有效的类别感知单次提示方法,通过提供类别内示例提升生成质量。对七种先进LLM的系统评估揭示:任务难度差异明显,对训练数据较少平台的泛化能力较差,且针对性提示策略有效。MultiKernelBench已开源,地址:https://github.com/wzzll123/MultiKernelBench。
原文摘要 · Abstract (English)
The automatic generation of deep learning (DL) kernels using large language models (LLMs) has emerged as a promising approach to reduce the manual effort and hardware-specific expertise required for writing high-performance operator implementations. However, existing benchmarks for evaluating LLMs in this domain suffer from limited hardware support, coarse-grained kernel categorization, and imbalanced task coverage. To address these limitations, we introduce MultiKernelBench, the first comprehensive, multi-platform benchmark for LLM-based DL kernel generation. MultiKernelBench spans 285 tasks across 14 well-defined kernel categories and supports three major hardware platforms: Nvidia GPUs, Huawei NPUs, and Google TPUs. To enable future extensibility, we design a modular backend abstraction layer that decouples platform-specific logic from the core benchmarking infrastructure, allowing easy integration of new hardware platforms. We further propose a simple yet effective category-aware one-shot prompting method that improves generation quality by providing in-category exemplars. Through systematic evaluations of seven state-of-the-art LLMs, we reveal significant variation in task difficulty, poor generalization to platforms with less training exposure, and the effectiveness of targeted prompting strategies. MultiKernelBench is publicly available at https://github.com/wzzll123/MultiKernelBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。