用语法探测法发现NL2SQL数据集的训练泄漏问题
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
- 设计语法变异测试框架SPENCE，系统检测模型对语序变化的敏感度
- 老数据集Spider在语法变化下准确率下降明显，新数据集BIRD几乎无影响
- 适合关注基准测试可信度的研究者和模型评估人员
大型语言模型（LLM）在自然语言转SQL（NL2SQL）基准上表现优异，但其报告的准确率可能因训练中见过相似查询或结构而被夸大。本文提出SPENCE（语法探测与NL2SQL污染效应评估），一种受控的语法探测框架，用于检测和量化此类污染。该框架系统生成四个主流数据集——Spider、SParC、CoSQL和BIRD——测试查询的语法变体。通过执行结果评分，评估多个高容量LLM在不同语法偏离程度下的表现变化，并使用肯德尔等级相关系数（Kendall's tau）与自助置信区间量化模型排名敏感性。结合基准发布日期分析，发现明显的时间梯度：较早发布的Spider等数据集呈现显著负相关，表明存在训练泄露；而较新的BIRD数据集敏感性极低，基本未受污染。结果强调了基于时间上下文的语法探测对可靠NL2SQL评估的重要性。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL datasets-Spider, SParC, CoSQL, and the newer BIRD benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall's tau with bootstrap confidence intervals. By aligning these robustness trends with benchmark release dates, we observe a clear temporal gradient: older benchmarks such as Spider exhibit the strongest negative values and thus the highest likelihood of training leakage, whereas the more recent BIRD dataset shows minimal sensitivity and appears largely uncontaminated. Together, these findings highlight the importance of temporally contextualized, syntactic-probing evaluation for trustworthy NL2SQL benchmarking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。