arXiv:2505.23804cs.CLcs.AI2025-05EMNLP被引 7

提升大模型文本转SQL的可信度,通过子句频率校准置信度。

Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies

  • 利用SQL结构化特性,统计子句出现频率作为校准信号。
  • 多变量Platt校准使置信度更准确,错误检测率提升12.3%。
  • 首次为文本转SQL建立后验校准基准,适合可靠性要求高的场景。

尽管大语言模型在文本转SQL任务中表现优异,但有时会自信地给出错误结果。构建可信的文本转SQL系统需要从模型获取可靠的不确定性度量。本文研究如何提供一个校准后的置信度分数,以反映输出查询正确的概率。我们的工作首次建立了基于LLM的文本转SQL解析的后验校准基准。我们发现,经典的Platt校准方法相比直接使用原始模型输出概率,显著提升了置信度质量。此外,我们提出一种利用SQL查询结构化特性的校准方法——子句频率(SCF)得分。通过多变量Platt校准(MPS),我们将多个SCF得分融合为一个整体且校准良好的置信度分数。在两个主流文本转SQL数据集上的实证评估表明,结合MPS与SCF的方法在校准性和错误检测任务上均优于传统Platt校准。

原文摘要 · Abstract (English)

While large language models (LLMs) achieve strong performance on text-to-SQL parsing, they sometimes exhibit unexpected failures in which they are confidently incorrect. Building trustworthy text-to-SQL systems thus requires eliciting reliable uncertainty measures from the LLM. In this paper, we study the problem of providing a calibrated confidence score that conveys the likelihood of an output query being correct. Our work is the first to establish a benchmark for post-hoc calibration of LLM-based text-to-SQL parsing. In particular, we show that Platt scaling, a canonical method for calibration, provides substantial improvements over directly using raw model output probabilities as confidence scores. Furthermore, we propose a method for text-to-SQL calibration that leverages the structured nature of SQL queries to provide more granular signals of correctness, named "sub-clause frequency" (SCF) scores. Using multivariate Platt scaling (MPS), our extension of the canonical Platt scaling technique, we combine individual SCF scores into an overall accurate and calibrated score. Empirical evaluation on two popular text-to-SQL datasets shows that our approach of combining MPS and SCF yields further improvements in calibration and the related task of error detection over traditional Platt scaling.

文本转SQL置信度校准大模型可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。