首个多语言代码效率评测基准,揭示大模型生成代码普遍低效。
EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
- 构建多语言代码效率评估框架,涵盖六种编程语言。
- 大模型平均仅达人类效率的62%,语言间差异显著。
- 适合关注代码优化与大模型工程落地的研究者。
现有代码生成评测主要关注功能正确性,对代码效率关注不足,且多局限于单一语言(如Python)。为此,我们提出EffiBench-X,首个支持Python、C++、Java、JavaScript、Ruby和Golang的多语言代码效率评测基准。该基准包含竞赛编程任务,并以人工专家解作为效率基线。对主流大模型的评估显示,尽管生成代码功能正确,但普遍低于人类专家效率。即使是最优模型Qwen3-32B,平均效率也仅达人类的62%,且存在明显语言差异:在Python、Ruby、JavaScript中表现优于Java、C++和Golang。例如,DeepSeek-R1在Python中的代码效率显著高于其在Java中的表现。这些结果凸显了提升大模型代码效率的迫切需求。数据集与评估工具已开源,地址为https://github.com/EffiBench/EffiBench-X.git 和 https://huggingface.co/datasets/EffiBench/effibench-x。
原文摘要 · Abstract (English)
Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around \textbf{62\%} of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。