用智能体评估生成的高性能计算代码,更全面发现模型漏洞。
An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
- 构建智能体框架,自动编译执行并评分代码。
- 模型生成代码虽可读但常违背库使用规范。
- 适合研究科学计算与AI代码质量的学者和开发者。
大型语言模型显著加速了科学计算代码生成,但全面评估生成代码仍面临挑战。传统基准仅依赖测试用例匹配,对高性能计算(HPC)中的库代码而言,求解器选择、API规范、内存管理与性能表现与功能正确性同等重要。为此,我们提出petscagent-bench框架,基于智能体评估智能体的范式,部署具备工具增强的评估智能体,对目标模型生成的代码进行编译、执行与测量,通过跨五类评分标准的14个评估环节实现综合评价:正确性、性能、代码质量、算法适配性及库特定规范。由于智能体间通过标准化协议(A2A与MCP)通信,该框架可黑盒评估任意编码智能体而无需源码访问。我们在使用PETSc库的真实问题基准上验证了该框架,实证分析表明当前前沿模型虽能生成结构清晰、可读性强的代码,但在库特定规范方面存在系统性缺陷,而传统通过/失败指标完全无法捕捉此类问题。
原文摘要 · Abstract (English)
While large language models have significantly accelerated scientific code generation, comprehensively evaluating the generated code remains a major challenge. Traditional benchmarks reduce evaluation to test-case matching, an approach insufficient for library code in HPC where solver selection, API conventions, memory management, and performance are just as critical as functional correctness. To address this gap, we introduce petscagent-bench, an agentic framework built on an agents-evaluating-agents paradigm. Instead of relying on static scripts, petscagent-bench deploys a tool-augmented evaluator agent that compiles, executes, and measures code produced by a separate model-under-test agent, orchestrating a 14-evaluator pipeline across five scoring categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. Because the agents communicate through standardized protocols (A2A and MCP), the framework enables black-box evaluation of any coding agent without requiring access to its source code. We demonstrate the framework on a benchmark suite of realistic problems using the PETSc library for HPC. Our empirical analysis of frontier models reveals that while current models generate readable, well-structured code, they consistently struggle with library-specific conventions that traditional pass/fail metrics completely miss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。