修复旧版WikiSQL缺陷，打造适配大模型的文本转SQL新基准。
LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
- 系统分类并自动化修正原数据集中的语法、类型等错误
- 10亿参数以下模型微调后准确率超90%，大模型零样本达88.4%
- 专为现代大模型设计，支持直接生成与评估
将自然语言问题转换为SQL查询，使非专家用户能与关系型数据库交互，是自然语言接口数据的核心任务。尽管早期文本转SQL研究依赖WikiSQL数据集，但其因大小写不一致、数据类型错配、语法错误及未解答问题等结构与标注缺陷，使用逐渐减少。本文提出LLMSQL，对WikiSQL进行系统性修订与重构，以适配大语言模型时代。我们分类识别各类错误，并实现自动化清洗与重标注。为评估改进效果，测试了包括Gemma 3、LLaMA 3.2、Mistral 7B、gpt-oss 20B、Phi-3.5 Mini、Qwen 2.5、OpenAI o4-mini、DeepSeek-R1等在内的多个大模型。结果显示，DeepSeek-R1在零样本设置下达到88.40%准确率，而参数量低于100亿的模型经微调后准确率超过90%。不同于简单更新，LLMSQL被定位为面向大模型的标准化基准。与原始版本仅适用于指针网络从输入中选择词元不同，本数据集提供干净的自然语言问题与完整纯文本SQL查询，便于现代自然语言转SQL模型直接生成与评估。
原文摘要 · Abstract (English)
Converting natural language questions into SQL queries enables non-expert users to interact with relational databases and has long been a central task for natural language interfaces to data. While the WikiSQL dataset played a key role in early text-to-SQL research, its usage has declined due to structural and annotation issues, including case sensitivity inconsistencies, data type mismatches, syntax errors, and unanswered questions. We present LLMSQL, a systematic revision and transformation of WikiSQL designed for the large language model era. We classify these errors and implement automated methods for cleaning and re-annotation. To assess the impact of these improvements, we evaluated multiple large language models, including Gemma 3, LLaMA 3.2, Mistral 7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek-R1, and others. Notably, DeepSeek-R1 achieves 88.40% accuracy in a zero-shot setting, and models under 10B parameters surpass 90% accuracy after fine-tuning. Rather than serving as an update, LLMSQL is introduced as an LLM-ready benchmark. Unlike the original WikiSQL, which was tailored for pointer-network models selecting tokens from input, LLMSQL provides clean natural language questions and full SQL queries as plain text, enabling straightforward generation and evaluation for modern natural-language-to-SQL models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。