首个评估大模型编写可执行PL/SQL程序的基准，揭示其在数据库编程中的真实能力
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
- 构建2865个任务，涵盖单轮与多轮对话，覆盖不同复杂度的数据库操作
- 八款大模型在语法正确性、流程控制和异常处理上表现普遍不足，尤其跨轮一致性差
- 适合关注数据库自动化、智能编程助手或大模型工程落地的研究者与开发者
我们提出PLSQLBench，据我们所知首个用于评估大模型是否能编写可执行PL/SQL程序的基准，通过执行测试衡量正确性。现有大模型评估多集中于通用代码生成或声明式文本转SQL，对过程式数据库编程关注不足。PLSQLBench包含2,865个实例：2,594个单轮任务和271个多轮对话，共978轮。基准融合了企业级Spider 2数据库的复杂模式相关任务、来自Spider的简单模式相关任务，以及基于MBPP的程序化问题，覆盖不同层次的数据库依赖性和过程复杂度。八款大模型实验显示，在模式理解、PL/SQL方言准确性、流程控制、异常处理及跨轮一致性方面存在持续困难。引入工具增强的大模型代理在部分模式相关任务中表现提升，但差距依然显著。结果表明，传统代码生成或文本转SQL基准无法充分评估过程式数据库编程能力。代码已开源：https://github.com/oracle-samples/plsqlbench。
原文摘要 · Abstract (English)
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。