针对企业级数据库设计了多层级评估基准，揭示现有NL2SQL模型在真实场景中的严重失效。
ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence
- 构建包含6个复杂企业的多数据库同步数据集，分三层次模拟真实企业场景。
- 模型在高复杂度层级上执行准确率降至57.2%，且多数正确执行结果语义错误。
- 适合关注企业级NL2SQL落地挑战的研究者与工程师阅读。
当前最先进的自然语言转SQL（NL2SQL）模型在Spider和BIRD等基准上的执行准确率超过89%。然而，这些基准依赖于简化的学术模式和开源SQL方言，无法反映企业数据库环境的复杂性。本文提出ESQ-Bench，一个以Oracle为核心的多层级企业级NL2SQL评估基准，涵盖三个企业模式复杂度层级，并引入沉默语义差异检测。我们构建并发布了六个已填充的数据库模式（共465张表、164,682行数据，无空表），在Oracle、PostgreSQL、MySQL和SQL Server上使用相同种子数据；设计四维评估指标（EM、EX、SR、SD）；提供550对经人工验证的问答-查询对（第1层：95个；第2层：228个；第3层：227个）。使用GPT-4o进行模式关联提示时，执行准确率（EX）随层级单调下降：79.8%、60.3%、57.2%（执行查询，2026年6月）；而早期142题测试集为75.6%、80.4%、95.8%。精确匹配（EM）全层级均低于7%；通过执行的查询中，操作性沉默语义偏差高达73%至99%。失败分析显示高复杂度下错误语义占主导。Claude Sonnet 4.6配合模式提示达到87.4%、74.9%、68.7%的执行准确率，优于所有层级的GPT-4o模式提示。GPT-4o零样本在执行查询上的准确率为78.7%、73.5%、77.8%，在第2至第3层反超模式提示，源于较低执行率与零样本分析中的幸存者偏差。本地Llama 3.2在模式提示下仅实现13.3%的银行级执行准确率（550条中有73条成功），凸显闭源大模型与开源基线在企业级Oracle模式上的巨大差距。
原文摘要 · Abstract (English)
State-of-the-art Natural Language to SQL (NL2SQL) models report execution accuracy exceeding 89 percent on established benchmarks such as Spider and BIRD. However, these benchmarks rely on simplified academic schemas and open-source SQL dialects that do not reflect the complexity of enterprise database environments. We introduce ESQ-Bench, an Oracle-first NL2SQL benchmark with systematic complexity tiers and silent-divergence evaluation across three enterprise schema complexity tiers. We constructed and released six populated schemas (465 tables, 164,682 rows, zero empty tables) with identical seed data on Oracle, PostgreSQL, MySQL, and SQL Server, a four-metric evaluation harness (EM, EX, SR, SD), and 550 gold-validated question-query pairs (Tier-1: 95; Tier-2: 228; Tier-3: 227). Schema-linked prompting with GPT-4o shows monotonic execution-match degradation across tiers: 79.8, 60.3, and 57.2 percent EX on executed queries (June 2026), versus 75.6, 80.4, and 95.8 percent on an earlier 142-question pilot slice. EM stays below 7 percent tier-wide; operational silent-divergence reaches 73 to 99 percent among EX-passing queries. Failure analysis shows wrong-result semantics dominate at higher tiers. Claude Sonnet 4.6 with schema-linked prompts reaches 87.4, 74.9, and 68.7 percent EX (executed queries), exceeding GPT-4o schema-linked on every tier. GPT-4o zero-shot EX on executed queries (78.7, 73.5, and 77.8 percent) inverts schema-linked at Tiers 2 to 3 due to lower execution rates and survivor bias in the zero-shot versus schema-linked analysis. Local Llama 3.2 schema-linked reaches only 13.3 percent bank-wide EX (73 out of 550), underscoring the gap between closed API models and open-weight baselines on enterprise Oracle schemas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。