开源模型通过智能训练框架，显著提升真实场景下SQL错误调试能力。
SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
- 构建自动回溯生成的SQL错误修复数据集，支持持续训练。
- 新方法使修复成功率在主流基准上达38.11%（PG）和29.65%（多方言）。
- 适合关注本地化数据库开发与隐私保护的研究者与工程师。
复杂SQL问题的解决仍是真实数据库应用中的主要瓶颈。现有大语言模型虽擅长文本转SQL，却未在更困难的SQL调试任务上得到充分评估。为此，我们提出BIRD-CRITIC基准，包含530个PostgreSQL任务（BIRD-CRITIC-PG）和570个多方言任务（BIRD-CRITIC-Multi），源自真实用户问题并重现在新环境以实现严格评估。基线测试显示，领先推理模型O3-Mini在BIRD-CRITIC-PG和BIRD-CRITIC-Multi上的成功率仅为38.87%和33.33%。为推动开源模型在数据库任务中的发展，保障数据隐私，我们提出Six-Gym（Sql-fIX-Gym）训练环境，采用SQL-Rewind策略自动从正确SQL反向生成可执行的错误-修复数据对。然而，主流轨迹微调方法未能充分挖掘监督信号。为此我们提出f-Plan Boosting，从解决方案中提取高层调试计划，使教师模型生成的可行轨迹增加73.7%。将这些组件集成至开源代理Bird-Fixer，基于Qwen-2.5-Coder-14B，在BIRD-CRITIC-PG和BIRD-CRITIC-Multi上分别取得38.11%和29.65%的成功率，超越Claude-3.7-Sonnet和GPT-4.1等专有模型，标志着向普及高级SQL调试能力迈出关键一步。排行榜与源码已公开：https://bird-critic.github.io/
原文摘要 · Abstract (English)
Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging SQL issues. To address this gap, we introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530 PostgreSQL tasks (BIRD-CRITIC-PG) and 570 multi-dialect tasks (BIRD-CRITIC-Multi), distilled from authentic user issues and replayed within new environments to facilitate rigorous evaluation. Baseline evaluations underscore the task's complexity, with the leading reasoning model O3-Mini achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi. Meanwhile, advancing open-source models for database tasks is crucial for empowering local development while safeguarding data privacy. Therefore, we present Six-Gym (Sql-fIX-Gym), a training environment for elevating open-source model capabilities for SQL issue debugging. This environment leverages SQL-Rewind strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose f-Plan Boosting, which extracts high-level debugging plans from SQL solutions, enabling teacher LLMs to produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, Bird-Fixer. Based on Qwen-2.5-Coder-14B, Bird-Fixer achieves 38.11% success rate on BIRD-CRITIC-PG and 29.65% on BIRD-CRITIC-Multi, surpassing leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities. The leaderboard and source code are available: https://bird-critic.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。