arXiv:2510.24358cs.SEcs.CL2025-10中稿 · AAMAS 2026被引 10

构建可扩展的代码智能体评估体系,提升真实项目级测试的准确性和多样性。

Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation

  • 用人类监督驱动生成多样化的项目级任务
  • 50个真实Python项目,每项有结构化需求文档和评估标准
  • 专用模型判官实现90%以上与人工评分一致

近期代码智能体的发展使大型语言模型(LLMs)支持项目级别的自动化软件开发。然而,现有代码智能体评估基准存在两大局限:一是创建高质量项目级数据集需大量领域知识,导致标注成本高昂且多样性不足;二是虽有以智能体为裁判的新范式缓解传统单元测试僵化问题,但依赖通用大模型的上下文学习(ICL)常导致评估结果偏离人工标准。为此,我们提出一种由智能体驱动的基准构建流程,借助人类监督高效生成多样化项目级任务。基于此,我们构建了PRDBench,包含50个涵盖20个领域的真实Python项目,每个项目配有结构化产品需求文档(PRDs)和全面评估标准。此外,为克服通用大模型判官的不准确性,我们设计了一个专用于评估的微调模型。基于Qwen3-Coder-30B的专用判官PRDJudge在固定接口场景下实现超过90%的人工对齐度。大量实验证明,本套件提供了可扩展、鲁棒且高度准确的前沿代码智能体评估框架。

原文摘要 · Abstract (English)

Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while recent Agent-as-a-Judge paradigms address the rigidity of traditional unit tests by enabling flexible metrics, their reliance on In-Context Learning (ICL) with general LLMs often results in inaccurate assessments that misalign with human standards. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks. Based on this, we introduce PRDBench, comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Documents (PRDs) and comprehensive criteria. Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model. Based on Qwen3-Coder-30B, our dedicated PRDJudge achieves over 90% human alignment in fixed-interface scenarios. Extensive experiments demonstrate that our suite provides a scalable, robust, and highly accurate framework for assessing state-of-the-art code agents.

代码生成智能体评估大模型评测真实项目

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。