首个全面评估大模型代码批判能力的基准，涵盖生成与问答任务。
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models
- 构建包含代码生成与问答的双任务评测集，覆盖不同难度。
- 设计细粒度评价清单，支持基础与进阶批判评估。
- 适合研究大模型代码理解与反馈能力的开发者与研究人员。
大型语言模型（LLM）的批判能力对推理至关重要，可提供详细分析与建设性反馈。现有批判性评测基准存在两大局限：一是聚焦通用领域推理任务，缺乏对代码任务的充分评估（如仅涵盖代码生成，且查询来自Humaneval和MBPP等简单数据集）；二是缺乏多维度综合评估。为解决这些问题，本文提出一个全面的代码批判评测基准CodeCriticBench。该基准包含两种主流代码任务（代码生成与代码问答），并涵盖不同难度。评估协议包括基础与进阶批判评估，进阶设置中设计了细粒度评价清单。我们对现有主流大模型进行了广泛实验，验证了CodeCriticBench的有效性。
原文摘要 · Abstract (English)
The critique capacity of Large Language Models (LLMs) is essential for reasoning abilities, which can provide necessary suggestions (e.g., detailed analysis and constructive feedback). Therefore, how to evaluate the critique capacity of LLMs has drawn great attention and several critique benchmarks have been proposed. However, existing critique benchmarks usually have the following limitations: (1). Focusing on diverse reasoning tasks in general domains and insufficient evaluation on code tasks (e.g., only covering code generation task), where the difficulty of queries is relatively easy (e.g., the code queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive evaluation from different dimensions. To address these limitations, we introduce a holistic code critique benchmark for LLMs called CodeCriticBench. Specifically, our CodeCriticBench includes two mainstream code tasks (i.e., code generation and code QA) with different difficulties. Besides, the evaluation protocols include basic critique evaluation and advanced critique evaluation for different characteristics, where fine-grained evaluation checklists are well-designed for advanced settings. Finally, we conduct extensive experimental results of existing LLMs, which show the effectiveness of CodeCriticBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。