新基准TIDE-Bench揭示对话式SQL中意图漂移与链歧义的挑战
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
- 构建包含1542样本的基准，聚焦意图演变中的层级澄清与请求替换
- 12个先进大模型在链识别上普遍表现差，意图漂移识别率低且修复能力弱
- 适合研究对话系统鲁棒性、数据库交互设计及多轮推理的学者使用
大语言模型（LLM）使对话式文本转SQL成为用户与数据库间的实用接口，通常涉及多轮澄清与修改。然而现有基准主要评估执行准确率，未充分揭示用户意图在多轮中的演变。为此，我们提出TIDE-Bench，一个针对链歧义与意图漂移的对话式文本转SQL评估基准。该基准基于BIRD中的514个锚点SQL，包含1,542个样本，引入链识别与漂移识别-修复的专用指标，超越传统执行准确率。对12个先进LLM的评估发现：链识别存在持续瓶颈，不受澄清频率影响；意图漂移识别与修复存在显著差距；当两种失败模式同时激活时，存在重叠。TIDE-Bench代码已开源，供后续研究使用。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。