评测大模型在真实企业级复杂SQL工作流中的表现，揭示其实际应用差距。
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
- 构建632个真实企业数据库场景的多查询、长上下文文本转SQL任务
- 模型仅解决21.3%任务，远低于以往基准（如Spider 1.0达91.2%）
- 适合研究智能代码代理、企业级AI工具与复杂推理能力的开发者
真实企业级文本转SQL工作流通常涉及跨云或本地数据库、多种SQL方言及从数据转换到分析的多样化操作。我们提出Spider 2.0，一个包含632个真实企业数据库用例的评估框架。这些数据库源自实际应用，常含超1000列，部署于BigQuery、Snowflake等系统。解决这些问题需理解数据库元数据、方言文档甚至项目级代码库。这要求模型具备与复杂环境交互、处理极长上下文、进行精细推理并生成超过100行的多步骤SQL查询，远超传统文本转SQL挑战。基于o1-preview的代码代理框架仅成功解决21.3%的任务，显著低于Spider 1.0的91.2%和BIRD的73.0%。结果表明，尽管大模型在代码生成上表现优异，但在真实企业场景中仍需大幅提升。推动Spider 2.0进展是实现真正智能自主代码代理的关键一步。代码、基线模型与数据已公开于https://spider2-sql.github.io。
原文摘要 · Abstract (English)
Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics. We introduce Spider 2.0, an evaluation framework comprising 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake. We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding 100 lines, which goes far beyond traditional text-to-SQL challenges. Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD. Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation -- especially in prior text-to-SQL benchmarks -- they require significant improvement in order to achieve adequate performance for real-world enterprise usage. Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings. Our code, baseline models, and data are available at https://spider2-sql.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。