arXiv:2412.05561cs.DBcs.LG2024-12被引 4

用大模型提升复杂SQL等价性检查,准确率超80%。

Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking

  • 设计新基准SQLEquiQuest,模拟真实复杂查询场景。
  • 大模型在Spider+DIN数据集上达82%准确率,覆盖率达100%。
  • 适合数据库研究者与自动化测试开发者参考。

SQL查询等价性检查是难以解决的问题,常见于评分、调试查询优化器等场景。尽管已有研究尝试提供实用方案,但仅支持少量基础查询,复杂查询仍依赖繁琐且易错的人工分析。本文探索大语言模型(LLM)在推理复杂SQL查询中的应用。为此,我们提出一个新颖、真实且足够复杂的基准SQLEquiQuest,反映实际使用环境。通过利用大模型对SQL的推理能力,建立强基线方法。我们对多个先进大模型进行评估,采用多种提示策略和精心构建的上下文学习示例,包括由查询处理器生成的逻辑计划。实证结果表明,大模型显著超越现有形式化模型,将支持查询对的比例从30%提升至100%,在Spider+DIN数据集上达到最高82%的准确率。然而,分析揭示大模型存在强烈的等价性偏好,对非等价查询对表现差,为未来研究指明新方向。

原文摘要 · Abstract (English)

Equivalence checking of SQL queries is an intractable problem often encountered in settings ranging from grading SQL submissions to debugging query optimizers. Despite recent work toward developing practical solutions, only simple queries written using a small subset of SQL are supported, leaving the equivalence checking of sophisticated SQL queries at the mercy of intensive, potentially error-prone, manual analysis. In this paper, we explore how LLMs can be used to reason with SQL queries to address this challenging problem. Towards this, we introduce a novel, realistic, and sufficiently complex benchmark called SQLEquiQuest for SQL query equivalence checking that reflects real-world settings. We establish strong baselines for SQL equivalence checking by leveraging the ability of LLMs to reason with SQL queries. We conduct a detailed evaluation of several state-of-the-art LLMs using various prompting strategies and carefully constructed in-context learning examples, including logical plans generated by SQL query processors. Our empirical evaluation shows that LLMs go well beyond the current capabilities of formal models for SQL equivalence, going from a mere 30% supported query pairs to full coverage, achieving up to 82% accuracy on Spider+DIN. However, a critical limitation of LLMs revealed by our analysis is that they exhibit a strong bias for equivalence predictions, with consistently poor performance over non-equivalent pairs, opening a new direction for potential future research.

SQL大模型等价检查

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。