首个基于企业私有数据的文本转SQL基准,揭示大模型在真实场景下表现严重不足。
BEAVER: An Enterprise Benchmark for Text-to-SQL
- 从真实企业查询日志构建9128对问答数据,覆盖19个领域
- 顶尖框架仅10.8%准确率,加提示后提升至30.1%
- 提供细粒度评估指标,适合研究企业级查询系统
现有文本转SQL基准多基于公共数据库,结构简单、问题单一。而大语言模型在复杂企业环境——如表结构复杂、需领域知识、查询涉及高级函数和嵌套结构——中的表现尚无验证。为此,我们提出BEAVER,首个基于私有数据仓库的文本转SQL基准。包含9128个真实查询日志生成的问题-SQL对,覆盖19个不同领域,共812张表。构建该基准面临两大挑战:一是企业查询日志因隐私限制难以获取;二是传统全或无准确率指标无法定位错误原因,尤其当问题涉及多个复合挑战时。我们从两个层面解决:数据层面,通过合成高保真、专家验证的查询,增加数据量并分离或组合特定挑战(如领域知识、查询复杂性);评估层面,引入人类标注和五项关键子任务的评估指标,实现细粒度分析。评估显示显著性能差距:最先进代理框架使用GPT-5.2仅达10.8%准确率。若提供所有子任务标注作为理想提示,准确率升至30.1%,证实主要瓶颈在于子任务的正确解析。最后,我们归纳出即使有提示仍存在的残余错误类型,如高级函数的误用。
原文摘要 · Abstract (English)
Existing text-to-SQL benchmarks have largely been constructed from public databases with well-structured schemas and simplistic question-SQL pairs. While large language models (LLMs) excel on these settings, their efficacy in complex private enterprise environments, characterized by intricate schemas, domain knowledge, and analytical user queries involving sophisticated structures and functions, remains unproven. To bridge this gap, we introduce BEAVER, the first text-to-SQL benchmark derived from private data warehouses. It comprises 9128 question-SQL pairs sourced from real-world query logs and 812 tables across 19 diverse domains. Building this benchmark is challenging because (1) enterprise query logs are scarce due to privacy constraints, and (2) existing all-or-nothing evaluation metrics based on accuracy make error diagnosis difficult -- especially when producing a correct query involves solving multiple compounded challenges, such as domain knowledge and query complexity. We address these issues at two levels. At the dataset level, we synthesize high-fidelity, expert-verified queries that increase dataset size and isolate individual challenges or combine them, producing queries focused on domain knowledge, query complexity, and both. At the evaluation level, we provide human annotations and evaluation metrics for five critical subtasks to enable fine-grained analysis. Our evaluation reveals a significant performance gap compared to existing benchmarks: SOTA agentic frameworks using the advanced model GPT-5.2 achieve only 10.8% accuracy. When provided with all subtask annotations as oracle hints, accuracy increases to 30.1%, confirming that a major bottleneck lies in correctly resolving these subtasks. Finally, we provide a taxonomy of the residual errors that persist even with subtask hints, identifying specific challenges such as the use of advanced functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。