首个面向复杂推理的Text-to-SQL基准,涵盖数学与假设推理场景。
LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning
- 构建包含45个数据库的链式思考数据集,覆盖物理、算术等复杂推理
- 4038个问题配12114步推理步骤,现有模型准确率最高仅33.20%
- 适合研究真实企业数据分析与自主查询生成的学者
Text-to-SQL是自然语言处理中的关键任务,旨在将自然语言问题转化为准确可执行的SQL查询。在真实场景中,此类推理常涉及复杂的数学计算、领域知识及假设性推理。然而,现有大规模Text-to-SQL数据集多聚焦于业务逻辑和任务逻辑,忽视了垂直领域知识、复杂数学推理与假设推理等关键因素,难以真实反映实际应用中的推理需求。为此,我们提出LogicCat,首个专为复杂推理与链式思考解析设计的Text-to-SQL基准数据集,涵盖物理、算术、常识与假设推理场景。LogicCat包含4,038个英文问题及12,114条详细链式思考推理步骤,覆盖45个跨领域数据库,显著超越现有数据集的复杂度。实验表明,LogicCat使当前最先进模型的执行准确率下降至最多33.20%,凸显该任务的极高挑战性。LogicCat的推出标志着向开发适用于真实企业数据分析与自主查询生成系统迈出了关键一步。数据集代码已开源:https://github.com/Ffunkytao/LogicCat。
原文摘要 · Abstract (English)
Text-to-SQL is a critical task in natural language processing that aims to transform natural language questions into accurate and executable SQL queries. In real-world scenarios, these reasoning tasks are often accompanied by complex mathematical computations, domain knowledge, and hypothetical reasoning scenarios. However, existing large-scale Text-to-SQL datasets typically focus on business logic and task logic, neglecting critical factors such as vertical domain knowledge, complex mathematical reasoning, and hypothetical reasoning, which are essential for realistically reflecting the reasoning demands in practical applications and completing data querying and analysis. To bridge this gap, we introduce LogicCat, the first Text-to-SQL benchmark dataset specifically designed for complex reasoning and chain-of-thought parsing, encompassing physics, arithmetic, commonsense, and hypothetical reasoning scenarios. LogicCat comprises 4,038 English questions paired 12,114 detailed chain-of-thought reasoning steps, spanning 45 databases across diverse domains, significantly surpassing existing datasets in complexity. Experimental results demonstrate that LogicCat substantially increases the task difficulty for current state-of-the-art models to at most 33.20% execution accuracy, indicating that this task remains exceptionally challenging. The advancement of LogicCat represents a crucial step toward developing systems suitable for real-world enterprise data analysis and autonomous query generation. We have released our dataset code at https://github.com/Ffunkytao/LogicCat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。