通过多路径推理与偏好优化,提升文本转SQL的准确率。
CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL
- 用分治法和思维链生成多样且高质量的SQL候选
- 在BIRD数据集上达到73.0%的执行准确率,领先当前方法
- 适合需要高精度文本转SQL的应用场景
针对大语言模型在文本转SQL任务中的性能挑战,我们提出CHASE-SQL框架,利用测试时计算资源,在多智能体建模中改进候选生成与选择。该框架通过不同语言模型生成器,结合三种创新策略:(1) 单次调用内将复杂查询分解为可管理子查询的分治法;(2) 基于查询执行计划的思维链推理,反映数据库引擎执行步骤;(3) 面向具体实例的合成示例生成技术,提供针对测试问题的特定少样本示范。为选出最优候选,采用一个选择代理,通过微调后的二元候选选择模型进行成对比较排序,证明其比其他方法更鲁棒。所提生成-选择框架不仅提升了SQL查询的质量与多样性,还超越先前方法。总体而言,CHASE-SQL在知名BIRD文本转SQL基准测试的测试集和开发集上分别达到73.0%和73.01%的执行准确率,成为提交时排行榜的最高分提交。
原文摘要 · Abstract (English)
In tackling the challenges of large language model (LLM) performance for Text-to-SQL tasks, we introduce CHASE-SQL, a new framework that employs innovative strategies, using test-time compute in multi-agent modeling to improve candidate generation and selection. CHASE-SQL leverages LLMs' intrinsic knowledge to generate diverse and high-quality SQL candidates using different LLM generators with: (1) a divide-and-conquer method that decomposes complex queries into manageable sub-queries in a single LLM call; (2) chain-of-thought reasoning based on query execution plans, reflecting the steps a database engine takes during execution; and (3) a unique instance-aware synthetic example generation technique, which offers specific few-shot demonstrations tailored to test questions.To identify the best candidate, a selection agent is employed to rank the candidates through pairwise comparisons with a fine-tuned binary-candidates selection LLM. This selection approach has been demonstrated to be more robust over alternatives. The proposed generators-selector framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods. Overall, our proposed CHASE-SQL achieves the state-of-the-art execution accuracy of 73.0% and 73.01% on the test set and development set of the notable BIRD Text-to-SQL dataset benchmark, rendering CHASE-SQL the top submission of the leaderboard (at the time of paper submission).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。