针对文本转SQL中大模型压缩难题，提出高效知识蒸馏方法KID。
Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL
- 用模拟推理误差的不完美数据提升蒸馏效果
- 在5个基准上平均性能提升最高达5.83%
- 无需额外训练成本，适合资源受限场景
大语言模型在文本转SQL任务中表现优异，但计算开销大，部署困难。知识蒸馏是压缩模型的常用方法，但现有自回归模型蒸馏方法在复杂文本转SQL场景中表现不佳，难以平衡性能与效率。为此，本文提出针对不完美数据的知识蒸馏方法KID，通过模拟推理过程中的级联误差，在不增加训练预算的情况下显著提升性能。在5个文本转SQL基准上的实验表明，KID在所有模型类型和规模下均实现一致且显著的性能提升（最高平均提升5.83%），同时有效提高训练效率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promising performance in text-to-SQL, which involves translating natural language questions into SQL queries. However, current text-to-SQL LLMs are computationally expensive and challenging to deploy in real-world applications, highlighting the importance of compressing them. To achieve this goal, knowledge distillation (KD) is a common approach, which aims to distill the larger teacher model into a smaller student model. While numerous KD methods for autoregressive LLMs have emerged recently, it is still under-explored whether they work well in complex text-to-SQL scenarios. To this end, we conduct a series of analyses and reveal that these KD methods generally fall short in balancing performance and efficiency. In response to this problem, we propose to improve the KD with Imperfect Data, namely KID, which effectively boosts the performance without introducing much training budget. The core of KID is to efficiently mitigate the training-inference mismatch by simulating the cascading effect of inference in the imperfect training data. Extensive experiments on 5 text-to-SQL benchmarks show that, KID can not only achieve consistent and significant performance gains (up to +5.83% average score) across all model types and sizes, but also effectively improve the training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。