LLMs生成移动端高效内核效果不佳,新系统提升成功率至93.7%。
MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?
- 构建移动端内核评估框架MobileKernelBench,支持跨框架验证。
- 现有LLMs编译失败率超54%,性能提升几乎为零。
- 提出MoKA多智能体系统,实现93.7%成功编译与27.4%加速。
大型语言模型(LLMs)在代码生成方面表现卓越,但其在移动端内核生成方面的潜力尚未被充分探索。本文将自动化内核生成拓展至移动端,提出核心问题:LLMs能否编写出高效的移动端内核?为此,我们构建了MobileKernelBench——一个注重算子多样性和跨框架互操作性的基准测试框架,并配备自动化流水线以解决主机-设备间差异。基于该框架,在MNN的CPU后端进行评估发现,当前主流LLMs因工程复杂性与数据稀缺性,面临超过54%的编译失败率,且因幻觉和领域知识缺失,性能提升微乎其微。为此,我们提出移动内核智能体(MoKA),采用仓库感知推理与计划-执行范式。在MobileKernelBench上验证,MoKA实现93.7%的编译成功率,并使27.4%的生成内核相比原生库获得可测量加速。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable capabilities in code generation, yet their potential for generating kernels specifically for mobile devices remains largely unexplored. In this work, we extend the scope of automated kernel generation to the mobile domain to investigate the central question: Can LLMs write efficient kernels for mobile devices? To enable systematic investigation, we introduce MobileKernelBench, a comprehensive evaluation framework comprising a benchmark prioritizing operator diversity and cross-framework interoperability, coupled with an automated pipeline that bridges the host-device gap for on-device verification. Leveraging this framework, we conduct extensive evaluation on the CPU backend of Mobile Neural Network (MNN), revealing that current LLMs struggle with the engineering complexity and data scarcity inherent to mobile frameworks; standard models and even fine-tuned variants exhibit high compilation failure rates (over 54%) and negligible performance gains due to hallucinations and a lack of domain-specific grounding. To overcome these limitations, we propose the Mobile Kernel Agent (MoKA), a multi-agent system equipped with repository-aware reasoning and a plan-and-execute paradigm. Validated on MobileKernelBench, MoKA achieves state-of-the-art performance, boosting compilation success to 93.7% and enabling 27.4% of generated kernels to deliver measurable speedups over native libraries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。