用大模型判断文本转SQL的正确性，发现推理类信号更有效。
What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
- 用验证器和自一致性信号评估SQL生成正确性，验证器表现更好。
- 多提供商模型集成可达到0.82 AUROC，实现低风险抽样。
- 通用跨模式验证仍需大模型推理能力，微调无法解决泛化问题。
评估AI生成SQL查询的不确定性，关键在于判断其是否正确——即执行结果与人工参考一致。研究在BIRD和Spider两个复杂多表文本转SQL数据集上，分析各类信号对正确性的预测能力，以AUROC衡量排序性能。黑箱信号如字符串、结构、执行自一致性、模式相关性评分和查询可执行性，表现均在0.61至0.68之间，其中字符串自一致性最强（0.675）；白箱对数概率表现相近（0.67）。真正突破这一上限的是基于验证的信号：不同提供者的LLM裁判评分从0.72（GPT-4o-mini）到0.78（Claude），且不同裁判犯错模式不同。双提供商集成达0.82 AUROC，校准误差仅0.03，并支持实用的弃权边界（如27%问题回答率下风险仅24%），而自一致性无法实现类似低风险子集。该规律在两基准、两生成器、两裁判提供商间一致。进一步探索可训练验证器，编码器与生成式微调模型在域内可达0.77~0.79 AUROC，但在未见模式上下降至约0.66；扩大至7B模型、增加模式多样性、蒸馏强裁判推理、跨基准训练均未能弥合差距。跨模式迁移表现与模型规模和推理能力相关，而非微调本身。实践中，文本转SQL的正确性不确定性依赖于推理型信号：微调验证器适用于特定场景，但泛化跨模式仍需大尺寸冻结推理模型。
原文摘要 · Abstract (English)
Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar (0.67). The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 (GPT-4o-mini) to 0.78 (Claude). Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability (expected calibration error 0.03) and supports useful abstention frontiers (for example, answering 27% of questions at 24% selective risk) where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。