为大模型文本转SQL设计动态交互评估基准，更贴近真实数据库应用挑战。
BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
- 构建可自主交互的仿真环境，支持澄清、纠错与知识检索。
- 在多轮对话中，顶级模型仅完成17%的任务，凸显复杂性。
- 适合研究智能数据库助手、具身智能与人机协同的学者。
大语言模型在单轮文本转SQL任务上表现优异，但真实场景需多轮交互以应对歧义、执行错误和需求演变。现有基准将对话历史视为静态上下文，或仅限只读操作，难以反映生产级数据库助手的真实挑战。本文提出BIRD-INTERACT，通过：（1）为每个数据库配置分层知识库、元数据文件与函数驱动的用户模拟器，使模型可在无监督下主动提问、查证信息并恢复错误；（2）提供预设协议（c-Interact）与开放自主代理设置（a-Interact），由模型自主决定何时询问或探索；（3）涵盖业务分析与运维全链条的增删改查任务，每项均含模糊及后续子任务，需动态交互。任务集包括600个任务（最多11,796次交互）的BIRD-INTERACT-FULL与300个简化数据库的BIRD-INTERACT-LITE。实测显示，GPT-5在c-Interact中仅完成8.67%，a-Interact中为17.00%。通过记忆嫁接与交互时缩放测试，验证了有效交互对复杂任务的关键作用。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short by treating conversation histories as static context or limiting evaluation to read-only operations, failing to reflect production-grade database assistant challenges. We introduce BIRD-INTERACT, a benchmark that restores this realism through: (1) a comprehensive interaction environment coupling each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from errors without human supervision; (2) two evaluation settings consisting of a pre-defined conversational protocol (c-Interact) and an open-ended agentic setting (a-Interact) where models autonomously decide when to query the user simulator or explore the environment; (3) a challenging task suite covering the full CRUD spectrum for business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600 tasks, up to 11,796 interactions) for comprehensive performance assessment, and BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed behavioral analysis and rapid method development. Our empirical results highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in c-Interact and 17.00% in a-Interact. Analysis via memory grafting and Interaction Test-time Scaling validates the importance of effective interaction for complex, dynamic text-to-SQL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。