用历史SQL日志提升文本转SQL能力,让模型学会理解隐含业务逻辑。
BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

- 基于历史查询日志提取五类外部知识,增强对模糊问题的理解。
- 在11个数据库上构建1393个任务,标注关键历史查询和对应代码片段。
- 可插拔检索器适配现有系统,无需改提示词即可提升准确率。
当前基于大语言模型的文本转SQL系统在标准基准上表现优异,但在用户查询依赖领域特定知识(如业务逻辑、数据规范、分析习惯)时表现不佳,这些知识既未包含在模式中,也未在自然语言中明确说明。历史SQL查询日志是此类知识的重要来源,但现有基准未能有效支持对历史驱动方法的评估。为此,我们提出BIRD-History,一个包含11个数据库、1393个任务的基准,用于评估文本转SQL系统利用历史SQL脚本来定位未明确定义的自然语言问题的能力。每个任务均标注了哪些历史查询包含相关知识,以及具体哪部分SQL语句编码了该知识,从而实现对知识检索与利用的系统性评估。同时,我们提出一种可插拔检索器,从历史SQL中提取五类外部知识,并检索与重排序相关片段以辅助生成查询。该检索器可无缝集成至现有少样本文本转SQL流程,无需修改提示词。实验表明,四种文本转SQL系统均获得一致性能提升,验证了利用历史查询日志处理未明确定义查询的价值。数据集与代码已开源于https://github.com/zjuidg/BIRD-History。
原文摘要 · Abstract (English)
While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。