用真实企业数据测试大模型生成SQL的准确性，发现复杂查询错误率高。
Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5
- 构建包含219个真实业务问题的评测集，覆盖5个难度等级。
- 简单聚合任务准确率达93%，复杂计算任务仅4%正确。
- 适合关注企业级AI数据查询可靠性的研发与评估人员。
大型语言模型（LLMs）在自然语言转结构化查询方面展现潜力，但在实际商业智能（BI）场景中受限于语义幻觉、结构错误及缺乏领域专用评估框架。本研究提出一种基于Exaone 3.5——一个针对企业任务优化的双语指令微调模型——的事实一致性评估框架，用于衡量生成SQL的语义准确性。我们构建了一个领域特定基准，包含219个来自LG电子内部BigQuery环境的真实销售数据的自然语言业务问题，涵盖五个SQL复杂度层级，每个问题配有标准SQL及验证后的真值答案。通过答案准确率、执行成功率、语义错误率和无响应率评估模型表现。实验表明，尽管Exaone 3.5在简单聚合任务（L1）中准确率达93%，但在复杂算术推理（H1）中降至4%，分组排名任务（H4）为31%，且语义错误与未响应集中于高复杂度案例。定性分析揭示常见错误包括算术逻辑误用、过滤不全和分组操作错误。研究凸显当前大模型在关键业务环境中的局限性，强调需引入事实一致性验证层与混合推理方法。本工作贡献可复现的基准与评估方法，推动企业级结构化数据自然语言接口的可靠性发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promise in enabling natural language interfaces for structured data querying through text-to-SQL generation. However, their application in real-world Business Intelligence (BI) contexts remains limited due to semantic hallucinations, structural errors, and a lack of domain-specific evaluation frameworks. In this study, we propose a Fact-Consistency Evaluation Framework for assessing the semantic accuracy of LLM-generated SQL outputs using Exaone 3.5--an instruction-tuned, bilingual LLM optimized for enterprise tasks. We construct a domain-specific benchmark comprising 219 natural language business questions across five SQL complexity levels, derived from actual sales data in LG Electronics' internal BigQuery environment. Each question is paired with a gold-standard SQL query and a validated ground-truth answer. We evaluate model performance using answer accuracy, execution success rate, semantic error rate, and non-response rate. Experimental results show that while Exaone 3.5 performs well on simple aggregation tasks (93% accuracy in L1), it exhibits substantial degradation in arithmetic reasoning (4% accuracy in H1) and grouped ranking tasks (31% in H4), with semantic errors and non-responses concentrated in complex cases. Qualitative error analysis further identifies common failure types such as misapplied arithmetic logic, incomplete filtering, and incorrect grouping operations. Our findings highlight the current limitations of LLMs in business-critical environments and underscore the need for fact-consistency validation layers and hybrid reasoning approaches. This work contributes a reproducible benchmark and evaluation methodology for advancing reliable natural language interfaces to structured enterprise data systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。