检测大模型生成SQL的结构可靠性，发现相同答案可能有多种写法。
SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation
- 用抽象语法树分析大模型生成SQL的结构差异
- 同一问题不同表达会导致结构不同，但执行正确率仍高
- 通过编译式流程可提升准确率和结构一致性
尽管大模型在Text-to-SQL基准上表现良好，但其生成的SQL程序在结构上是否可靠仍不明确。本文研究了大模型生成SQL的结构行为，提出SQLStructEval框架，通过规范的抽象语法树（AST）表示分析程序结构。在Spider基准上的实验表明，现代大模型对同一输入常生成结构多样的查询，即使执行结果正确，这种差异也常由输入表面变化（如改写或模式展示方式）触发。进一步证明，采用编译式流水线在结构化空间生成查询，可同时提升执行准确率与结构一致性。结果表明，结构可靠性是评估大模型程序生成系统时被忽视的关键维度。代码已公开于https://anonymous.4open.science/r/StructEval-2435。
原文摘要 · Abstract (English)
Despite strong performance on Text-to-SQL benchmarks, it remains unclear whether LLM-generated SQL programs are structurally reliable. In this work, we investigate the structural behavior of LLM-generated SQL queries and introduce SQLStructEval, a framework for analyzing program structures through canonical abstract syntax tree (AST) representations. Our experiments on the Spider benchmark show that modern LLMs often produce structurally diverse queries for the same input, even when execution results are correct, and that such variance is frequently triggered by surface-level input changes such as paraphrases or schema presentation. We further show that generating queries in a structured space via a compile-style pipeline can improve both execution accuracy and structural consistency. These findings suggest that structural reliability is a critical yet overlooked dimension for evaluating LLM-based program generation systems. Our code is available at https://anonymous.4open.science/r/StructEval-2435.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。