无需标注数据即可评估新文本转SQL模型的准确率
An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
- 利用模型输出模式分析数据差异，推断未标注数据的准确率
- 在多种场景下预测真实准确率，误差小于5%
- 适合部署后持续监控和发布前质量检查
大语言模型的进步提升了文本转SQL系统的能力，但新训练的模型在未见过且无标注的数据集上评估仍面临挑战。由于数据库内容和结构不断变化，隐私政策限制人工审核，且精确的SQL标签成本高昂，导致无法及时评估。FusionSQL解决了这一问题，适用于任意文本转SQL模型，可在无参考答案情况下估算准确率。它通过分析模型自身输出的模式，识别目标数据集与训练数据的差异。该方法支持发布前检测、新数据库的持续监控及性能退化预警。在多种应用场景和问题类型下的实验表明，FusionSQL能紧密跟随实际准确率，并可靠发现潜在问题。代码已开源：https://github.com/phkhanhtrinh23/FusionSQL。
原文摘要 · Abstract (English)
Recent advances in large language models have strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。