arXiv:2606.04023cs.SEcs.AI2026-06

评测大模型在多种硬件上生成高效代码的能力,发现其跨平台表现不均。

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

论文配图:CodegenBench: Can LLMs Write Efficient Code Across Architectures?
图 1 · 摘自论文原文
  • 构建跨x86_64、Sunway、Kunpeng三架构的代码生成评测集
  • 大模型在主流x86_64上表现良好,在特定架构上性能显著下降
  • 适合研究高性能计算与大模型结合的开发者参考

尽管大语言模型(LLMs)已在通用编程和GPU加速环境(如PyTorch、CUDA)的代码生成任务中得到广泛评估,但其在多样硬件平台上面向CPU的高性能计算(HPC)中的表现仍鲜有研究。为此,我们提出CodegenBench,一个涵盖x86_64、Sunway和Kunpeng三个硬件平台的综合性基准测试套件。该套件包含106个标准基本线性代数子程序(BLAS)例程作为基础基准,并针对每种超算架构定制了20个专用计算内核(LeetSunway和LeetKunpeng)。评估结果显示,当前最先进的LLMs虽能在普遍的x86_64架构上生成优化代码,但在缺乏公开文档与训练数据的专用架构上表现明显退化,暴露出跨平台泛化能力的严重不足。此外,分析表明代码质量受实现长度与任务复杂度影响,模型在中等难度、需简洁代码的场景下效果最佳。我们已开源数据集与自动化评估工具,链接为https://anonymous.4open.science/r/CodegenBench-EDE1/ 和 https://anonymous.4open.science/r/CodegenBenchDataset-2551。

原文摘要 · Abstract (English)

While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.g., PyTorch, CUDA), their capabilities in CPU-oriented high-performance computing (HPC) across diverse architectures remain underexplored. To bridge this gap, we introduce CodegenBench, a comprehensive benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms: x86_64, Sunway, and Kunpeng. Our benchmark comprises 106 standard Basic Linear Algebra Subprograms (BLAS) routines establishing a fundamental baseline, alongside 20 specialized computational kernels adapted for each of the unique supercomputing architectures (LeetSunway and LeetKunpeng). Our extensive evaluation reveals that while state-of-the-art LLMs can generate optimized code for ubiquitous architectures like x86_64, they exhibit significant performance degradation on domain-specific architectures with limited public documentation and training data, highlighting critical limitations in cross-platform generalization. Furthermore, our analysis of factors influencing code quality such as implementation length and task complexity indicates that current LLMs are most effective for moderately difficult problems requiring concise code snippets. We open-source our dataset and automated evaluation infrastructure to facilitate future research in LLM-driven high-performance code generation. The resources are available at https://anonymous.4open.science/r/CodegenBench-EDE1/ and https://anonymous.4open.science/r/CodegenBenchDataset-2551.

代码生成大模型高性能计算跨架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。