提出跨方言SQL评估新方法，解决现有基准仅限SQLite的局限性
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
- 通过双执行比对归一化结果，无需手动转换SQL语句
- 在所有方言上实现100%覆盖，准确率平均下降10.1%
- 揭示逻辑错误是主要问题，适合需方言鲁棒性的研究者
SQL方言在语法、类型和函数上存在差异，但现有文本转SQL基准大多仅支持SQLite，导致跨方言评估存在重大缺口：单查询一致性低（Cohen's κ），表明SQLite性能无法代表其他方言。现有方法或需昂贵的手动转换，或依赖易失败的工具。为此，我们提出PolySQL，一种无需查询转换的双执行方法，通过比较归一化执行结果实现高保真评估，达到100%查询覆盖率。该框架包含三个数据集，首次支持大规模跨方言研究。结果显示，从SQLite到其他方言平均准确率下降10.1%，且存在显著方言难度层级。分析发现，61%的错误源于逻辑问题而非语法（8%）。我们开源了框架代码与排行榜，推动方言鲁棒评估的发展。
原文摘要 · Abstract (English)
SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL benchmarks, however, predominantly support only SQLite. This creates a critical evaluation gap: cross-dialect evaluation reveals weak per-query agreement (Cohen's ), showing that SQLite performance is an unreliable proxy for other dialects. Yet such evaluation remains prohibitively difficult: existing approaches either require expensive manual query transpilation or rely on tools that often fail on complex SQL. To close this gap, we introduce PolySQL, a novel dual-execution method that eliminates the need for query transpilation by comparing normalized execution results. Notably, our approach achieves higher evaluation fidelity than query transpilation with 100% query coverage. PolySQL comprises three datasets, enabling the first large-scale cross-dialect study. Our study reveals a 10.1% average accuracy drop from SQLite to other dialects and identifies a significant dialect difficulty hierarchy. We find this degradation stems from logical rather than syntactic errors (61% vs. 8%). We release our framework code and leaderboard to enable rigorous dialect-robust evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。