提出新评估框架，揭示文本转SQL中查询的语义结构与分布规律
Anatomy of a Query: W5H Dimensions and FAR Patterns for Text-to-SQL Evaluation
- 基于FAR结构不变性，所有查询可分解为过滤、聚合、返回三类操作
- 医疗领域查询80.4%聚焦时间维度，73%涉及人物信息，因果推理几乎缺失
- 适用于评估语言模型对数据库的深层理解能力，尤其适合医疗场景研究者
自然语言数据库接口日益流行，但其评估与设计的理论基础仍不完善。本文提出QUEST（通过语义转换进行查询理解评估）框架，包含两个独立构建的组件：FAR结构不变性，即每个有效查询均可归约为筛选、聚合、返回操作；以及W5H维度框架，认为所有筛选条件均映射至六个语义维度（谁、什么、哪里、何时、为何、如何）。在五个文本转SQL数据集（n=120,464）上验证表明，FAR一致性在所有领域和模式类型中普遍成立，而W5H维度分布差异显著。医疗查询在时间（WHEN: 80.4%）和人物中心（WHO: 73.0%）维度高度集中，远超通用基准；因果（WHY）和机制（HOW）推理在各领域均接近零，少数看似“如何”的例外实为数量聚合，非真正过程推理。这些发现揭示了迈向真实机器结构化数据推理所需跨越的前沿。
原文摘要 · Abstract (English)
Natural language interfaces to databases have gained popularity, yet the theoretical foundations for evaluating and designing these systems remain underdeveloped. We present QUEST (Query Understanding Evaluation through Semantic Translation), a framework resting on two independently motivated components: the FAR structural invariant, which holds that every well-formed query reduces to Filter, Aggregate, and Return operations; and the W5H dimensional framework, which holds that all filtering criteria map to six semantic dimensions (Who, What, Where, When, Why, and How). Validated across five text-to-SQL datasets (n = 120,464), FAR conformance is universal across all domains and schema types, while W5H dimensional profiles vary substantially. Healthcare queries are strongly concentrated in temporal (WHEN: 80.4%) and person-centric (WHO: 73.0%) dimensions far exceeding general-domain benchmarks, and causal (WHY) and mechanistic (HOW) reasoning are near-zero everywhere, with apparent HOW exceptions reflecting quantitative aggregation rather than genuine procedural reasoning. These results identify a frontier that must be crossed for genuine machine reasoning over structured data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。