构建首个包含模糊与无解问题的对话式文本转SQL数据集,提升模型真实场景理解能力。
PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries
- 构建含四类模糊和四类无解问题的对话数据,模拟真实用户提问
- 设计四轮对话流程,支持澄清与多角度响应,提升系统鲁棒性
- 发现当前顶级模型在处理模糊/无解问题上表现不佳,适合研究真实场景下NLP系统
以往文本转SQL数据集和系统主要关注意图明确且可回答的问题。但真实用户提问常存在歧义或因缺乏相关数据而无法回答。本文构建了一个名为PRACTIQ的实用对话式文本转SQL数据集,包含受真实用户提问启发的模糊与无解问题。通过分析现有数据集,我们识别出四类模糊问题和四类无解问题,并设计四轮对话结构:初始问题、助手请求澄清、用户澄清、助手生成修正后的SQL及执行结果自然语言解释。对于部分模糊问题,也直接生成考虑多重歧义的有益SQL响应,而非仅请求澄清。为评估模型在模糊、无解与可回答问题上的表现,我们使用多种大语言模型(LLM)实现基线系统,方法包括问题类别分类与澄清型SQL预测两步。实验表明,当前先进系统在处理模糊与无解问题时仍表现不佳。代码与数据将开源于GitHub。
原文摘要 · Abstract (English)
Previous text-to-SQL datasets and systems have primarily focused on user questions with clear intentions that can be answered. However, real user questions can often be ambiguous with multiple interpretations or unanswerable due to a lack of relevant data. In this work, we construct a practical conversational text-to-SQL dataset called PRACTIQ, consisting of ambiguous and unanswerable questions inspired by real-world user questions. We first identified four categories of ambiguous questions and four categories of unanswerable questions by studying existing text-to-SQL datasets. Then, we generate conversations with four turns: the initial user question, an assistant response seeking clarification, the user's clarification, and the assistant's clarified SQL response with the natural language explanation of the execution results. For some ambiguous queries, we also directly generate helpful SQL responses, that consider multiple aspects of ambiguity, instead of requesting user clarification. To benchmark the performance on ambiguous, unanswerable, and answerable questions, we implemented large language model (LLM)-based baselines using various LLMs. Our approach involves two steps: question category classification and clarification SQL prediction. Our experiments reveal that state-of-the-art systems struggle to handle ambiguous and unanswerable questions effectively. We will release our code for data generation and experiments on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。