构建首个跨数据库SQL翻译基准，测试大模型对不同系统语法的理解能力。
PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation
- 设计真实场景下的跨数据库SQL翻译数据集，覆盖22个生产级系统。
- 大模型平均准确率低于38.53%，凸显对系统特有语法理解的不足。
- 提供多样与简化两个版本，适合不同强度的模型压力测试。
大型语言模型在文本转SQL任务中表现日益出色，但另一重要问题——跨系统SQL翻译（即SQL-to-SQL），将针对某一数据库系统（如MySQL）编写的查询转换为另一系统（如ClickHouse）的等价形式——仍缺乏深入研究。现有基准普遍局限于少数数据库（常仅SQLite），且难以涵盖系统特有的SQL方言（如自定义函数、数据类型和语法规则）。为此，本文提出PARROT：一个面向跨系统SQL翻译的实用且真实的基准。PARROT包含来自38个开源基准和真实业务服务的598组翻译对，专为挑战系统特有语法理解而设计，大模型平均准确率不足38.53%。同时提供多个变体：PARROT-Diverse含28,003条翻译，用于广泛语法测试；PARROT-Simple含5,306个代表性样本，用于聚焦压力测试。为推动后续研究，我们公开了排行榜与源代码：https://code4db.github.io/parrot-bench/。
原文摘要 · Abstract (English)
Large language models (LLMS) have shown increasing effectiveness in Text-to-SQL tasks. However, another closely related problem, Cross-System SQL Translation (a.k.a., SQL-to-SQL), which adapts a query written for one database system (e.g., MySQL) into its equivalent one for another system (e.g., ClickHouse), is of great practical importance but remains underexplored. Existing SQL benchmarks are not well-suited for SQL-to-SQL evaluation, which (1) focus on a limited set of database systems (often just SQLite) and (2) cannot capture many system-specific SQL dialects (e.g., customized functions, data types, and syntax rules). Thus, in this paper, we introduce PARROT, a Practical And Realistic BenchmaRk for CrOss-System SQL Translation. PARROT comprises 598 translation pairs from 38 open-source benchmarks and real-world business services, specifically prepared to challenge system-specific SQL understanding (e.g., LLMS achieve lower than 38.53% accuracy on average). We also provide multiple benchmark variants, including PARROT-Diverse with 28,003 translations (for extensive syntax testing) and PARROT-Simple with 5,306 representative samples (for focused stress testing), covering 22 production-grade database systems. To promote future research, we release a public leaderboard and source code at: https://code4db.github.io/parrot-bench/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。