给大模型生成的SQL查语句打可信度分，提升可靠性。
Confidence Estimation for Text-to-SQL in Large Language Models
- 用查询一致性判断可信度，黑盒也能用。
- 结合语法感知能更好解读模型输出逻辑。
- 执行结果可作额外依据，适合部署验证场景。
文本到SQL的置信度估计旨在不依赖标准答案的情况下评估模型生成的SQL查询的可靠性。本文研究在大型语言模型（LLM）背景下该问题，其中模型权重和梯度通常不可访问。我们探索了黑盒与白盒两种置信度估计策略，并在跨领域文本到SQL基准上评估其有效性。结果表明，一致性方法在黑盒设置中表现更优，而基于SQL语法感知的方法在白盒设置下更能有效解释模型logits。此外，我们发现查询的执行结果可作为有价值的补充信号，显著提升两类方法的效果。
原文摘要 · Abstract (English)
Confidence estimation for text-to-SQL aims to assess the reliability of model-generated SQL queries without having access to gold answers. We study this problem in the context of large language models (LLMs), where access to model weights and gradients is often constrained. We explore both black-box and white-box confidence estimation strategies, evaluating their effectiveness on cross-domain text-to-SQL benchmarks. Our evaluation highlights the superior performance of consistency-based methods among black-box models and the advantage of SQL-syntax-aware approaches for interpreting LLM logits in white-box settings. Furthermore, we show that execution-based grounding of queries provides a valuable supplementary signal, improving the effectiveness of both approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。