用人类与大模型协作，高效生成高质量Text-to-SQL数据。
Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation
- 设计人机协同流程，分步生成领域适配的Text-to-SQL数据。
- 相比人工标注和纯AI生成，数据准确率更高、多样性更好。
- 适合需快速部署到新数据库场景的开发者与工程师。
Text-to-SQL模型将自然语言问题转化为可执行的SQL查询，在实际应用中日益普及。然而，将其部署到具体应用时，常面临专用数据库模式带来的性能显著下降，主要因缺乏领域特定数据用于微调。现有数据稀缺也限制了对新领域模型性能的有效评估。持续获取高质量的Text-to-SQL数据以适应不断演进的数据库模式，在真实场景中成本过高。为此，我们提出SQLsynth——一种人机协同的文本到SQL数据标注系统。该系统通过结构化工作流实现人类与大模型的高效协作，显著提升数据生成效率。在一项针对同一用户群体的对照研究中，对比人工标注和ChatGPT，SQLsynth在标注速度、认知负荷、数据准确性、自然度和多样性方面均表现更优。代码已开源：https://github.com/magic-YuanTian/SQLsynth。
原文摘要 · Abstract (English)
Text-to-SQL models, which parse natural language (NL) questions to executable SQL queries, are increasingly adopted in real-world applications. However, deploying such models in the real world often requires adapting them to the highly specialized database schemas used in specific applications. We find that existing text-to-SQL models experience significant performance drops when applied to new schemas, primarily due to the lack of domain-specific data for fine-tuning. This data scarcity also limits the ability to effectively evaluate model performance in new domains. Continuously obtaining high-quality text-to-SQL data for evolving schemas is prohibitively expensive in real-world scenarios. To bridge this gap, we propose SQLsynth, a human-in-the-loop text-to-SQL data annotation system. SQLsynth streamlines the creation of high-quality text-to-SQL datasets through human-LLM collaboration in a structured workflow. A within-subjects user study comparing SQLsynth with manual annotation and ChatGPT shows that SQLsynth significantly accelerates text-to-SQL data annotation, reduces cognitive load, and produces datasets that are more accurate, natural, and diverse. Our code is available at https://github.com/magic-YuanTian/SQLsynth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。