用SQL生成自然语言来测试NL2SQL模型对表达变化的鲁棒性。
Evaluating NL2SQL via SQL2NL
- 基于SQL2NL生成语义相同但表达多样的查询，控制变量评估鲁棒性。
- 顶尖模型在改写查询上准确率下降超10%，大模型也显著退化。
- 适合关注真实场景泛化能力的开发者与评估研究者参考。
在存在语言变异的情况下进行稳健评估，是理解自然语言转SQL（NL2SQL）模型泛化能力的关键，但现有基准很少以系统化、受控的方式处理这一因素。我们提出一种新型的模式对齐改写框架，利用SQL到自然语言（SQL2NL）自动生成语义等价且词汇多样的查询，同时保持与原始模式和意图的一致性。这实现了首次在孤立条件下针对NL2SQL对语言变异的鲁棒性评估——区别于以往主要研究歧义或模式扰动的工作。分析表明，当前最先进模型的鲁棒性远低于基准测试所暗示的程度。例如，LLaMa3.3-70B在改写后的Spider查询上执行准确率从77.11%降至66.9%，下降10.23%；而LLaMa3.1-8B下降近20%（从62.9%至42.5%）。较小模型（如GPT-4o mini）受影响更严重。我们还发现，鲁棒性退化随查询复杂度、数据集和领域显著变化，凸显了需要显式衡量语言泛化的评估框架，以确保实际应用中的可靠性能。
原文摘要 · Abstract (English)
Robust evaluation in the presence of linguistic variation is key to understanding the generalization capabilities of Natural Language to SQL (NL2SQL) models, yet existing benchmarks rarely address this factor in a systematic or controlled manner. We propose a novel schema-aligned paraphrasing framework that leverages SQL-to-NL (SQL2NL) to automatically generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. This enables the first targeted evaluation of NL2SQL robustness to linguistic variation in isolation-distinct from prior work that primarily investigates ambiguity or schema perturbations. Our analysis reveals that state-of-the-art models are far more brittle than standard benchmarks suggest. For example, LLaMa3.3-70B exhibits a 10.23% drop in execution accuracy (from 77.11% to 66.9%) on paraphrased Spider queries, while LLaMa3.1-8B suffers an even larger drop of nearly 20% (from 62.9% to 42.5%). Smaller models (e.g., GPT-4o mini) are disproportionately affected. We also find that robustness degradation varies significantly with query complexity, dataset, and domain -- highlighting the need for evaluation frameworks that explicitly measure linguistic generalization to ensure reliable performance in real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。