评测并提升大模型在多轮对话中处理多种问题类型的能力
Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types
- 构建多轮多类型问题测试集MMSQL，模拟真实对话场景
- 发现主流大模型在模糊问题上表现不佳，准确率低于60%
- 提出多智能体框架，按问题类型选择策略，显著提升鲁棒性
大语言模型在文本转SQL任务中取得显著进展，但多数方法仅关注SQL生成，忽视真实对话中复杂的多轮交互与多样化问题类型。这导致对模糊问题响应不可靠。为此，我们提出MMSQL，一个全面的评估套件，通过模拟真实场景中的多轮问答和多种问题类型，评估大模型在问题分类与SQL生成方面的能力。利用MMSQL，我们评估了包括开源与闭源在内的多个主流大模型，并识别出影响其性能的关键因素。此外，我们引入基于大模型的多智能体框架，通过专用智能体识别问题类型并制定应对策略。实验表明，该方法显著增强了模型处理对话复杂性的能力，有效应对用户查询的多样性和复杂性。数据集与代码已公开于https://mcxiaoxiao.github.io/MMSQL。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries. Our dataset and code are publicly available at https://mcxiaoxiao.github.io/MMSQL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。