让AI在不确定时主动放弃,提升文本转SQL的可靠性。
Reliable Text-to-SQL with Adaptive Abstention
- 引入自适应拒答机制,关键步骤中自动检测错误
- 在BIRD数据集上实现近完美的模式链接准确率
- 适合需要高可靠性的数据库自然语言接口场景
大型语言模型(LLMs)已革新数据库的自然语言接口,尤其在文本转SQL任务中表现突出。然而,面对模糊或上下文不足的情况,现有方法常产生不可靠输出。本文提出可靠的文本转SQL框架RTS,通过引入拒答与人机协作机制提升查询生成的可靠性。RTS聚焦于关键的模式链接阶段,自主检测生成过程中的潜在错误,并选择拒答或触发用户交互。其核心组件分支点预测(BPP)利用大模型隐藏层的统计置信区间技术,为模式链接提供概率保证。在BIRD基准测试中,该方法显著提升鲁棒性与可靠性,实现近乎完美的模式链接准确率,在必要时可自主调用人工干预。结合小型查询生成模型,其整体性能几乎媲美使用规模大数个数量级的顶尖模型。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized natural language interfaces for databases, particularly in text-to-SQL conversion. However, current approaches often generate unreliable outputs when faced with ambiguity or insufficient context. We present Reliable Text-to-SQL (RTS), a novel framework that enhances query generation reliability by incorporating abstention and human-in-the-loop mechanisms. RTS focuses on the critical schema linking phase, which aims to identify the key database elements needed for generating SQL queries. It autonomously detects potential errors during the answer generation process and responds by either abstaining or engaging in user interaction. A vital component of RTS is the Branching Point Prediction (BPP) which utilizes statistical conformal techniques on the hidden layers of the LLM model for schema linking, providing probabilistic guarantees on schema linking accuracy. We validate our approach through comprehensive experiments on the BIRD benchmark, demonstrating significant improvements in robustness and reliability. Our findings highlight the potential of combining transparent-box LLMs with human-in-the-loop processes to create more robust natural language interfaces for databases. For the BIRD benchmark, our approach achieves near-perfect schema linking accuracy, autonomously involving a human when needed. Combined with query generation, we demonstrate that near-perfect schema linking and a small query generation model can almost match SOTA accuracy achieved with a model orders of magnitude larger than the one we use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。