arXiv:2510.24762cs.CLcs.AI2025-10被引 1

面向企业级场景的中文文本转SQL评测基准,挑战多表关联与业务语义理解。

Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation

  • 基于真实企业数据架构构建跨领域中文SQL生成测试集。
  • 当前最先进模型在该基准上准确率最高仅50%,暴露关键缺陷。
  • 适合研究中文语义解析、企业级数据库系统与智能查询的学者。

我们提出Falcon,一个基于企业级方言(MaxCompute/Hive)的跨领域中文文本转SQL评测基准,包含600个中文问题,覆盖28个数据库;77%的问题需多表推理,超过一半涉及四个以上表的连接。每个样本均标注了SQL计算特征与中文语义信息。评估方面,我们发布了鲁棒的执行比对器和自动化评估流水线,所有现有主流大模型(包括Deepseek)在此基准上的准确率均不超过50%。主要错误源于两大挑战:(1) 大型企业环境中模式链接困难——数百张表、非规范化字段、模糊列名、隐式外键关系及领域特定同义词导致正确连接/列选择复杂;(2) 将简洁口语化中文精确映射为分析所需操作符与谓词——如选择正确的聚合与分组键、表达时间窗口与粒度、执行单位转换、处理空值与数据质量规则、构造嵌套或窗口子查询。Falcon因此聚焦中文特有语义与企业方言(缩写、业务术语、模糊实体引用),通过真实企业模式、查询模板、执行比对器与自动化评估流水线,提供可复现的生产前验证中间地带。

原文摘要 · Abstract (English)

We introduce Falcon, a cross-domain Chinese text-to-SQL benchmark grounded in an enterprise-compatible dialect (MaxCompute/Hive). It contains 600 Chinese questions over 28 databases; 77% require multi-table reasoning and over half touch more than four tables. Each example is annotated along SQL-computation features and Chinese semantics. For evaluation, we release a robust execution comparator and an automated evaluation pipeline, under which all current state-of-the-art large-scale models (including Deepseek) achieve accuracies of at most 50%. Major errors originate from two sources: (1) schema linking in large enterprise landscapes - hundreds of tables, denormalized fields, ambiguous column names, implicit foreign-key relations and domain-specific synonyms that make correct join/column selection difficult; and (2) mapping concise, colloquial Chinese into the exact operators and predicates required for analytics - e.g., choosing the correct aggregation and group-by keys, expressing time windows and granularities, applying unit conversions, handling NULLs and data-quality rules, and formulating nested or windowed subqueries. Falcon therefore targets Chinese-specific semantics and enterprise dialects (abbreviations, business jargon, fuzzy entity references) and provides a reproducible middle ground before full production deployment by using realistic enterprise schemas, query templates, an execution comparator, and an automated evaluation pipeline for end-to-end validation.

文本转SQL中文NLP企业级数据评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。