用智能体框架评估复杂代码生成,更准更透明。
CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
- 设计双阶段智能体框架,自动提取多维上下文信息。
- 在363个复杂代码样本上,相关性指标平均提升0.14以上。
- 适合需要高精度、可解释性代码评估的研究者和开发者。
大型语言模型在代码生成方面表现出色,亟需严谨全面的评估方法。现有评估方式分为人工、基于指标和基于LLM三类;其中人工方式耗时,基于指标的方式依赖参考答案,而基于LLM的方法通常仅基于静态提示,难以应对涉及多重需求的复杂代码场景,且缺乏细粒度分析与可解释性。为此,本文提出首个面向复杂代码生成的智能体评估框架CodeVisionary,包含两阶段:(1)需求引导的多维度上下文提炼阶段,(2)细粒度评分与摘要生成阶段,并输出综合评估报告以增强可解释性。我们构建了一个包含363个样本、覆盖37种编程场景和23种编程语言的新基准。大量实验表明,CodeVisionary在评估复杂代码生成方面优于三个基线,在皮尔逊、斯皮尔曼和肯德尔τ相关系数上分别平均提升0.217、0.163和0.141。相关资源已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong capabilities in code generation, underscoring the critical need for rigorous and comprehensive evaluation. Existing evaluation approaches fall into three categories, including human-centered, metric-based, and LLM-based. Considering that human-centered approaches are labour-intensive and metric-based ones overly rely on reference answers, LLM-based approaches are gaining increasing attention due to their stronger contextual understanding capabilities. However, they generally evaluate the generated code based on static prompts, and tend to fail for complex code scenarios which typically involve multiple requirements and require more contextual information. In addition, these approaches lack fine-grained evaluation for complex code, resulting in limited explainability. To mitigate the limitations, we propose CodeVisionary, the first agent-based evaluation framework for complex code generation. CodeVisionary consists of two stages: (1) Requirement-guided multi-dimensional context distillation stage and (2) Fine-grained scoring and summarization stage. A comprehensive evaluation report is also generated for enhanced explainability. For validation, we construct a new benchmark consisting of 363 samples spanning 37 coding scenarios and 23 programming languages. Extensive experiments demonstrate that CodeVisionary achieves the best performance among three baselines for evaluating complex code generation, outperforming the best baseline with average improvements of 0.217, 0.163, and 0.141 in Pearson, Spearman, and Kendall-Tau coefficients, respectively. The resources of CodeVisionary are available at https://github.com/Eshe0922/CodeVisionary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。