用小模型逼近大模型的文本转SQL能力,提升效率。
SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

- 通过大模型生成合成数据,指导小模型学习。
- 在WikiSQL上达到86.9%执行准确率,接近大模型表现。
- 适合资源受限场景下的高效文本转SQL应用。
文本转SQL是自然语言处理中的基础任务,使用户能以自然语言与结构化数据库交互。尽管大型语言模型(LLMs)在此任务上表现出色,但其巨大的计算需求限制了在资源受限环境中的部署。本文提出SQuaD-SQL(Small-Qualified and Distilled for SQL),一种新方法,通过LLM引导的知识蒸馏和合成数据生成,使小型语言模型(SLMs)在文本转SQL任务上接近LLMs性能的同时显著提升效率。该方法包含三个关键组件:(1) 基于LLM的合成数据生成,通过精心设计的提示策略从LLMs中提取结构化知识;(2) 参数高效微调,可在单块消费级GPU上完成全模型训练;(3) 领域自适应微调,利用领域特定合成数据进一步提升目标领域的性能。在WikiSQL数据集上的实验表明,SQuaD-SQL在测试集上达到86.9%的执行准确率,接近大模型表现,同时具备更快推理速度和更低内存占用。结果表明,通过合适的训练策略,SLMs可成为资源受限环境中文本转SQL应用的实际高效替代方案。
原文摘要 · Abstract (English)
Text-to-SQL is a fundamental task in natural language processing that enables users to interact with structured databases using natural language. While large language models (LLMs) have demonstrated remarkable performance on this task, their substantial computational requirements hinder deployment in resource-constrained settings. In this paper, we introduce SQuaD-SQL (Small-Qualified and Distilled for SQL), a novel approach that empowers small language models (SLMs) to approach the performance of LLMs on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation. Our method comprises three key components: (1) LLM-based synthetic data generation, where structured knowledge is extracted from LLMs via carefully designed prompting strategies; (2) parameter-efficient fine-tuning, enabling full model training on a single consumer-grade GPU; and (3) domain-adaptive fine-tuning, where domain-specific synthetic data further enhances performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieves an execution accuracy of 86.9% on the test set, approaching the performance of LLMs while offering faster inference and lower memory usage. These results suggest that, with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。