多语言Text-to-SQL新基准揭示大模型跨语言能力短板
Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- 构建8语种文本转SQL基准MultiSpider 2.0，融合语言与方言差异
- 主流大模型在新基准上执行准确率仅4%，远低于旧版60%
- 引入协作式语言代理迭代优化查询，准确率提升至15%
Text-to-SQL实现自然语言访问数据库，但现有评测多局限于英语，制约多语言发展。我们提出MultiSpider 2.0，将Spider 2.0扩展至八种语言（英语、德语、法语、西班牙语、葡萄牙语、日语、中文、越南语），在保留原结构复杂度的基础上，增加语言与方言多样性，要求更深层次推理生成复杂SQL。在该基准上，最先进的大语言模型（如DeepSeek-R1和OpenAI o1）仅靠内在推理达到4%的执行准确率，而此前在MultiSpider 1.0上可达60%。为此，我们提出一种协作驱动的语言代理基线，通过迭代优化查询，将准确率提升至15%。结果揭示了显著的多语言差距，推动开发跨语言鲁棒性更强、适合企业部署的方法。基准数据集已开源：https://github.com/phkhanhtrinh23/Multilingual_Text_to_SQL。
原文摘要 · Abstract (English)
Text-to-SQL enables natural access to databases, yet most benchmarks are English-only, limiting multilingual progress. We introduce MultiSpider 2.0, extending Spider 2.0 to eight languages (English, German, French, Spanish, Portuguese, Japanese, Chinese, Vietnamese). It preserves Spider 2.0's structural difficulty while adding linguistic and dialectal variability, demanding deeper reasoning for complex SQL. On this benchmark, state-of-the-art LLMs (such as DeepSeek-R1 and OpenAI o1) reach only 4\% execution accuracy when relying on intrinsic reasoning, versus 60\% on MultiSpider 1.0. Therefore, we provide a collaboration-driven language agents baseline that iteratively refines queries, improving accuracy to 15\%. These results reveal a substantial multilingual gap and motivate methods that are robust across languages and ready for real-world enterprise deployment. Our benchmark is available at https://github.com/phkhanhtrinh23/Multilingual_Text_to_SQL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。