SpotIt+通过约束验证提升文本转SQL评估的准确性
SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints
- 基于数据库约束搜索差异实例，增强评估严谨性
- 在BIRD数据集上生成更贴近实际的反例数据库
- 适合评估文本转SQL系统的真实性能，尤其关注错误检测
我们提出SpotIt+，一个开源工具，通过有界等价验证评估文本转SQL系统。给定生成的SQL查询与标准答案，SpotIt+主动搜索能区分两者的数据库实例。为确保生成的反例反映实际差异，引入一种尽力而为的约束挖掘流程，结合规则式规范挖掘与大模型在示例数据库上的验证。在BIRD数据集上的实验表明，挖掘出的约束使SpotIt+生成更真实的差异化数据库，同时保持高效发现标准测试评估遗漏的大量查询差异的能力。
原文摘要 · Abstract (English)
We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a best-effort constraint-mining pipeline that combines rule-based specification mining with LLM-based validation over example databases. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。