评测大模型生成高性能系统代码的能力,发现其效率远低于专家水平。
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

- 构建可执行的性能优化基准,要求硬件感知与并行优化。
- 大模型生成代码在GPU任务上效率差距超50倍,且难以稳定达标。
- 适合关注系统级代码优化与模型性能评估的研究者。
大型语言模型(LLMs)虽能生成功能正确的代码,但在高性能系统任务的效率优化方面能力有限。现有代码基准多关注正确性或算法求解,缺乏对真实系统级优化的考察。为此,我们提出PerfCodeBench,一个用于评估大模型在系统级代码优化能力的可执行基准。任务涵盖系统实现选择、硬件感知优化及性能瓶颈处理,每项任务包含可执行正确性检测、基线实现和参考优化方案,支持对正确性与运行时效率的双重评估。我们在一系列先进大模型上进行评估,发现模型生成代码与专家优化实现之间存在显著差距,尤其在并行化和GPU操作任务中,性能差距可达50倍以上。当前模型还暴露出跨语言鲁棒性差、难以持续达到专家级效率的问题。结果表明,仍需引入以性能为导向的评估机制,推动大模型从仅生成正确代码向高效系统软件生成演进。数据、评估框架与所有模型生成代码已公开于https://anonymous.4open.science/r/perfcodebench-7CDE。
原文摘要 · Abstract (English)
Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution. This allows us to evaluate both correctness and runtime-oriented efficiency. Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations. Current models also show weaknesses in cross-language robustness and in consistently reaching expert-level efficiency. These results suggest that performance-aware evaluation are still needed. LLMs should move beyond generating merely correct code toward producing efficient systems software. We submit the benchmark data, evaluation infrastructure, and complete logs of all LLMs-generated code at https://anonymous.4open.science/r/perfcodebench-7CDE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。