给大模型加语义文档,能显著提升数据查询准确率
Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
- 用4KB语义文档补充数据库说明,降低模型误判
- 准确率提升17至23个百分点,最高达68.7%
- 适合做数据查询的LLM应用开发与评估
将大语言模型用于自然语言查询分析型数据库时,常出现错误答案和自信幻觉,根源在于模型需推断模式中未编码的业务语义。我们测试在输入中加入这些语义是否能缩小差距。在ClickHouse上的Cleaned Contoso Retail Dataset上,对三款前沿模型(Claude Opus 4.7、Claude Sonnet 4.6、GPT-5.4)进行100个自然语言问题的配对单次评测。每模型两次测试:一次仅提供仓库模式,一次额外加入4 KB手写Markdown文档,描述度量、约定与歧义规则。加入文档后,所有模型准确率提升17至23个百分点。有文档时,三模型表现统计上无差异(67.7%-68.7%);无文档时亦无差异(45.5%-50.5%)。跨集群比较均在p < 0.01水平显著。语义层文档解释了几乎全部显著方差,同层级模型选择无关紧要。这表明:显式业务语义并非增强模型能力,而是改变了任务本质。
原文摘要 · Abstract (English)
LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause: the model is forced to infer business semantics that the schema does not encode. We test whether supplying those semantics as context closes the gap. We benchmark three frontier LLMs (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4) on 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse, using a paired single-shot protocol. Each model is evaluated twice: once given only the warehouse schema, and once given the schema plus a 4 KB hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. Adding the document improves accuracy by +17 to +23 percentage points across all three models. With it, the three models are statistically indistinguishable (67.7-68.7%); without it, they are also indistinguishable (45.5-50.5%). Every cross-cluster comparison is significant at p < 0.01. The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not. We interpret this as a structural result: explicit business semantics suppress the dominant class of text-to-SQL errors not by making the model more capable, but by changing what the model is being asked to do.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。