让大模型像人一样从经验中学习,持续提升文本转SQL能力。
Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL
- 构建可扩展的经验知识库,通过记录成功与失败案例实现无参数持续学习。
- 小模型Llama-3.1-70B性能超越大模型Llama-3.1-405B的SOTA方法。
- 适合需要长期优化、低成本迭代的文本转SQL应用开发者使用。
大型语言模型在多项任务中表现出色,但在文本转SQL等下游应用中仍远逊于人类表现。在BIRD基准上,人类准确率为92.96%,而最优方法仅为72.39%。现有先进方法主要依赖上下文学习模拟人类推理,却忽略了人类关键能力——持续学习。受“错题本”教育实践启发,我们提出LPE-SQL框架,通过可扩展的辅助知识库实现无需参数微调的持续学习。该框架包含四个模块:检索相关条目、高效生成SQL、通过交叉一致性机制输出最终结果,以及记录成功/失败任务及其推理过程或反思提示。核心在于第四模块,其余模块采用基础方法,便于与现有技术融合。实验表明,该持续学习策略带来显著性能提升,使较小的Llama-3.1-70B模型超越使用SOTA方法的大模型Llama-3.1-405B。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit impressive problem-solving skills across many tasks, but they still underperform compared to humans in various downstream applications, such as text-to-SQL. On the BIRD benchmark leaderboard, human performance achieves an accuracy of 92.96\%, whereas the top-performing method reaches only 72.39\%. Notably, these state-of-the-art (SoTA) methods predominantly rely on in-context learning to simulate human-like reasoning. However, they overlook a critical human skill: continual learning. Inspired by the educational practice of maintaining mistake notebooks during our formative years, we propose LPE-SQL (Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL), a novel framework designed to augment LLMs by enabling continual learning without requiring parameter fine-tuning. LPE-SQL consists of four modules that \textbf{i)} retrieve relevant entries, \textbf{ii)} efficient sql generation, \textbf{iii)} generate the final result through a cross-consistency mechanism and \textbf{iv)} log successful and failed tasks along with their reasoning processes or reflection-generated tips. Importantly, the core module of LPE-SQL is the fourth one, while the other modules employ foundational methods, allowing LPE-SQL to be easily integrated with SoTA technologies to further enhance performance. Our experimental results demonstrate that this continual learning approach yields substantial performance gains, with the smaller Llama-3.1-70B model with surpassing the performance of the larger Llama-3.1-405B model using SoTA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。