arXiv:2511.13984cs.LGcs.AI2025-11被引 2

通过分析SQL语法树节点,精准定位大模型生成代码的错误位置。

Node-Level Uncertainty Estimation in LLM-Generated SQL

  • 基于语义感知标签算法,为语法树每个节点打标正确性。
  • 用特征分类器预测节点错误概率,平均AUC提升27.44%。
  • 适合需要精准调试、自动修复或人工审核的SQL生成系统。

我们提出一种实用框架,通过估计查询抽象语法树(AST)中各节点的不确定性,检测大语言模型生成的SQL中的错误。该方法分两阶段:首先设计一种语义感知的标签算法,给生成的SQL与标准答案比对时,不因结构容器或别名差异而过度惩罚;其次,为每个节点构建包含模式感知和词法特征的丰富表示,涵盖标识符有效性、别名解析、类型兼容性、作用域歧义及拼写错误信号,并训练监督分类器预测节点级错误概率。这些概率被解释为校准后的不确定性,实现细粒度诊断,精确定位查询可能出错的位置。在多个数据库和数据集上,该方法显著优于词元置信度:平均AUC提升27.44%,且跨数据库评估中仍保持稳健。该不确定性不仅作为准确率信号,还可支持针对性修复、人机协同审查与下游选择性执行。结果表明,以节点为中心、语义基础的不确定性估计,是传统序列级置信度的更优、可解释替代方案。

原文摘要 · Abstract (English)

We present a practical framework for detecting errors in LLM-generated SQL by estimating uncertainty at the level of individual nodes in the query's abstract syntax tree (AST). Our approach proceeds in two stages. First, we introduce a semantically aware labeling algorithm that, given a generated SQL and a gold reference, assigns node-level correctness without over-penalizing structural containers or alias variation. Second, we represent each node with a rich set of schema-aware and lexical features - capturing identifier validity, alias resolution, type compatibility, ambiguity in scope, and typo signals - and train a supervised classifier to predict per-node error probabilities. We interpret these probabilities as calibrated uncertainty, enabling fine-grained diagnostics that pinpoint exactly where a query is likely to be wrong. Across multiple databases and datasets, our method substantially outperforms token log-probabilities: average AUC improves by +27.44% while maintaining robustness under cross-database evaluation. Beyond serving as an accuracy signal, node-level uncertainty supports targeted repair, human-in-the-loop review, and downstream selective execution. Together, these results establish node-centric, semantically grounded uncertainty estimation as a strong and interpretable alternative to aggregate sequence level confidence measures.

SQL生成不确定性估计大模型可解释性代码诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。