arXiv:2510.26495cs.DBcs.CL2025-10被引 1

构建动态多轮文本转SQL评测基准,提升真实场景下数据库交互能力。

Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

  • 基于数据库结构自动合成多轮交互任务,覆盖13个领域。
  • GPT-4o在该基准上准确率仅58.34%,Pass@5为23.81%。
  • 适用于评估模型在真实业务场景中的持续迭代查询能力。

当前文本转SQL研究在静态单轮任务中表现优异,但在真实交互场景中仍显不足,用户需根据中间结果不断调整查询条件。为此,我们提出DySQL-Bench,一个通过自动化两阶段流程(任务合成与验证)构建的多轮交互评测基准。利用原始数据库表生成的结构化树引导大模型生成任务,并经交互导向筛选与专家验证,人工评估确认合成数据100%正确。我们设计了多轮评估框架,模拟大模型用户、待测模型与可执行数据库之间的动态交互。模型需随用户意图变化持续调整推理与SQL生成。该基准涵盖BIRD与Spider 2数据库的13个领域,共1,072个任务。即使最强模型GPT-4o也仅达58.34%整体准确率和23.81% Pass@5,凸显其挑战性。所有代码与数据已开源。

原文摘要 · Abstract (English)

Recent advances in Text-to-SQL have achieved strong results in static, single-turn tasks, where models generate SQL queries from natural language questions. However, these systems fall short in real-world interactive scenarios, where user intents evolve and queries must be refined over multiple turns. In applications such as finance and business analytics, users iteratively adjust query constraints or dimensions based on intermediate results. To evaluate such dynamic capabilities, we introduce DySQL-Bench, a benchmark assessing model performance under evolving user interactions. Unlike previous manually curated datasets, DySQL-Bench is built through an automated two-stage pipeline of task synthesis and verification. Structured tree representations derived from raw database tables guide LLM-based task generation, followed by interaction-oriented filtering and expert validation. Human evaluation confirms 100% correctness of the synthesized data. We further propose a multi-turn evaluation framework simulating realistic interactions among an LLM-simulated user, the model under test, and an executable database. The model must adapt its reasoning and SQL generation as user intents change. DySQL-Bench covers 13 domains across BIRD and Spider 2 databases, totaling 1,072 tasks. Even GPT-4o attains only 58.34% overall accuracy and 23.81% on the Pass@5 metric, underscoring the benchmark's difficulty. All code and data are released at https://github.com/Aurora-slz/Real-World-SQL-Bench .

文本转SQL多轮交互数据库探索评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。