首个面向偏微分方程求解器自动生成的多指标、多库评测基准。
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation

- 构建跨6类数学问题、11种方程类型的645个实例,覆盖3个主流FEM库。
- 采用分阶段评估框架,要求代码通过可执行性、精度与效率三重验证。
- 揭示当前大模型生成代码在精度和效率上严重不足,适合数值模拟研究者使用。
偏微分方程到求解器代码的自动生成旨在从方程描述自动合成可执行的数值求解器。该任务不仅需理解PDE的数学结构,还需选择合适的离散化方案与求解器配置,并在有限元方法(FEM)库中正确实现。现有代码生成评测主要关注语法正确性或预设测试用例的成功率。据我们所知,目前尚无公开的专用评测基准用于PDE-to-solver代码生成,通用代码评测也未能充分反映数值求解的独特挑战,如求解精度、效率及与专业FEM库的兼容性。本文提出PDEAgent-Bench,据我们所知是首个多指标、多库的评测基准。该基准包含645个实例,覆盖6个数学类别和11种PDE类型,支持DOLFINx、Firedrake和deal.II三个常用FEM库。每个实例提供面向智能体的问题描述、指定网格上的参考解,以及具体的精度和运行时间目标。评测采用分阶段流程:生成的求解器必须依次通过可执行性、数值精度和计算效率检查。对代表性LLMs和代码智能体的实验表明,模型虽常能生成可运行代码,但一旦引入精度与效率要求,通过率显著下降。结果表明,当前智能体在生成数值可靠且高效的求解器方面仍存在明显局限,而PDEAgent-Bench为实际数值求解需求提供了可复现的测试平台。
原文摘要 · Abstract (English)
PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. This task requires not only understanding the mathematical structure of PDEs, but also selecting appropriate discretization schemes and solver configurations, and correctly implementing the resulting formulations in finite-element method (FEM) libraries. Existing code generation benchmarks mainly evaluate syntactic correctness, or success on predefined test cases. To our knowledge, there is currently no publicly available benchmark specifically for PDE-to-solver code generation, and general-purpose code benchmarks do not fully capture the unique challenges of numerical PDE solution, such as ensuring solver accuracy, efficiency, and compatibility with professional FEM libraries. We introduce PDEAgent-Bench, to the best of our knowledge, the first multi-metric, multi-library benchmark for PDE-to-solver code generation. PDEAgent-Bench contains 645 instances across 6 mathematical categories and 11 PDE families, with common FEM libraries for DOLFINx, Firedrake, and deal.II. Each instance provides an agent-facing problem specification, a reference solution on a prescribed evaluation grid, and case-specific accuracy and runtime targets. PDEAgent-Bench adopts a staged evaluation framework in which generated solvers must sequentially pass executability, numerical accuracy, and computational efficiency checks. Experiments with representative LLMs and code agents show that models can often produce runnable code, but their pass rate drops substantially once accuracy and efficiency requirements are enforced. These results indicate that current agents remain limited in producing numerically reliable and efficient PDE solvers, and that PDEAgent-Bench provides a reproducible testbed grounded in the practical requirements of numerical PDE solving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。