用编程高尔夫挑战评估大模型写简洁代码能力，发现推理模型表现更好。
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
- 基于编程高尔夫竞赛设计多语言代码压缩评测基准。
- 推理模型在Python和C++上平均得分70.97%，远超非推理模型。
- 适合关注代码效率与大模型编程能力的研究者使用。
本文提出CodeGolf Bench，一个可在60种编程语言中评估大语言模型（LLMs）生成简洁代码能力的基准。该基准基于代码高尔夫——一种以最少字符或字节实现为目标的趣味编程竞赛——提供对模型生成高效、紧凑代码能力的独特衡量。不同于现有基准受限于固定题目集和语言覆盖范围，CodeGolf Bench利用code.golf平台获取新题目并实时获取人类表现基线。在Python和C++任务上的九个LLM评估显示，推理模型显著优于非推理模型，最佳平均百分位达70.97%。这一差距在语法严格的C++中尤为明显，凸显推理能力对复杂语言的重要性。非推理模型在两语言中均难以优化效率，最佳百分位显著低于推理模型。CodeGolf Bench为评估大模型代码生成能力提供了动态框架，可随人类表现演变而持续更新。
原文摘要 · Abstract (English)
This paper introduces Code Bench, a benchmark capable of evaluating Large Language Models (LLMs) concise code generation abilities in 60 programming languages. Based on code golf, a recreational programming competition focused on minimal character or byte solutions, the benchmark provides a distinctive measure of LLMs ability to produce efficient, concise code. Unlike existing benchmarks limited by fixed problem sets and language coverage, CodeGolf Bench leverages the code.golf platform to provide new problems and live human performance baselines. Evaluation of nine LLMs on Python and C++ tasks demonstrates that reasoning models significantly outperform non-reasoning models, achieving best average percentile of 70.97%. This performance gap is particularly pronounced in C++, highlighting reasoning's importance for languages with strict syntax requirements. Non-reasoning models struggle more with efficiency optimization across both languages, with best percentiles significantly lower than reasoning counterparts. CodeGolf Bench offers a dynamic framework for evaluating LLM code generation capabilities against evolving human performance on code golf.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。