新基准CORGI测试复杂商业查询,推动大模型从查数据到做决策。
Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain
- 构建企业级合成数据库,涵盖四类递进式商业问题
- 大模型在高阶任务中执行成功率下降33.12%
- 适合研究智能代理与开放生成评估的学者
传统文本转SQL基准仅测试简单数据查询,而真实用户常提出需预测或推荐的复杂问题。本文以商业场景为例,提出CORGI新基准,包含受DoorDash、Airbnb、Lululemon启发的合成数据库,覆盖描述性、解释性、预测性和推荐性四类递进式业务查询。该任务要求因果推理、时间预测与策略建议,体现多层级多步骤智能体能力。实验表明,大模型在更高阶问题上性能显著下降:相比BIRD等基准,平均执行成功率(SER)降低33.12%。同时,本工作倡导对开放式定性回答的自动评估方法。我们公开发布CORGI数据集、评估框架及提交平台,支持后续研究。
原文摘要 · Abstract (English)
Text-to-SQL benchmarks have traditionally only tested simple data access as a translation task of natural language to SQL queries. But in reality, users tend to ask diverse questions that require more complex responses including data-driven predictions or recommendations. Using the business domain as a motivating example, we introduce CORGI, a new benchmark that expands text-to-SQL to reflect practical database queries encountered by end users. CORGI is composed of synthetic databases inspired by enterprises such as DoorDash, Airbnb, and Lululemon. It provides questions across four increasingly complicated categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal reasoning, temporal forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance degrades on higher-level questions as question complexity increases. CORGI also introduces and encourages the text-to-SQL community to consider new automatic methods for evaluating open-ended, qualitative responses in data access tasks. Our experiments show that LLMs exhibit an average 33.12% lower success execution rate (SER) on CORGI compared to existing benchmarks such as BIRD, highlighting the substantially higher complexity of real-world business needs. We release the CORGI dataset, an evaluation framework, and a submission website to support future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。