arXiv:2601.18119cs.AI2026-01被引 1

首个企业级SQL调试基准,测试大模型真实场景下的纠错能力

Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

  • 用逆向工程自动生成带真实错误的复杂SQL,支持大规模测试
  • 顶尖模型在语法错误上仅36.5%准确率,语义错误更低至32.2%
  • 适合数据工程、LLM应用研究者,关注生产环境代码可靠性

SQL是企业数据工程的核心,但即使对经验丰富的开发者和先进文本转SQL的大语言模型而言,一次生成完全正确的SQL仍极困难,常需多次调试。我们提出OurBench,首个面向企业级SQL推理与调试的基准。该基准基于两项关键创新:(1) 采用逆向工程的自动化构建流程,系统性地在大规模SQL代码中注入真实错误,实现可扩展且多样化的基准生成;(2) 设计面向企业场景的无执行评估框架,提供快速、精准且资源高效的评估方式。OurBench包含469个OurBenchSyn查询(含语法错误及明确报错信息),以及516个OurBenchSem查询(针对未能满足用户意图的语义错误)。这些查询高度复杂,平均超过140行,抽象语法树深度与广度均显著。对近30个LLM的评估显示显著性能差距:表现最佳的Claude-4-Sonnet在OurBenchSyn上仅达36.46%准确率,在OurBenchSem上为32.17%,多数模型低于20%。我们进一步探索四种解决方案策略,识别关键挑战,并提出未来企业级SQL调试中使用LLM的可行方向。

原文摘要 · Abstract (English)

SQL is central to enterprise data engineering, yet generating fully correct SQL code in a single attempt remains difficult, even for experienced developers and advanced text-to-SQL LLMs, often requiring multiple debugging iterations. We introduce OurBench, the first benchmark for enterprise-level SQL reasoning and debugging. Our benchmark is built on two key innovations: (1) an automated construction workflow that uses reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an execution-free evaluation framework tailored to enterprise settings, providing fast, accurate, and resource-efficient assessment. OurBench comprises 469 OurBenchSyn queries featuring syntax errors with explicit error messages, and 516 OurBenchSem queries targeting semantic errors in which the code fails to meet user intent. The queries are highly complex, averaging over 140 lines and featuring deep and wide abstract syntax trees. Evaluation of nearly 30 LLMs reveals a substantial performance gap: the best-performing model, Claude-4-Sonnet, achieves only 36.46 percent accuracy on OurBenchSyn and 32.17 percent on OurBenchSem, while most models score below 20 percent. We further explore four solution strategies, identify key challenges, and outline promising directions for enterprise SQL debugging with LLMs.

SQL生成大模型评测数据工程代码调试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。