无需数据库模式和标准答案,实时评估生产环境中文本转SQL的准确率。
Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems
- 基于自然语言和生成SQL提取语义规范,进行无模式匹配评估
- 输出0-100分可解释的准确率,支持过滤、排序等容错处理
- 适合需要持续优化生产级文本转SQL系统的团队使用
生产环境中文本转SQL(T2SQL)评估面临根本性挑战,现有基准无法应对真实场景。当前方法依赖真实查询和结构化数据库模式,而这些在实际部署中通常不可得。这导致生产中的T2SQL代理缺乏有效评估,质量悄然下降却无反馈机制。本文提出无模式文本转SQL评估框架STEF,仅需用户问题、增强重述和生成的SQL,无需数据库模式或参考查询。STEF从自然语言和SQL中提取语义规范,进行归一化特征对齐,通过融合过滤对齐、语义判断和评估器置信度的综合指标,输出0至100的可解释准确率。关键贡献包括:将问题质量验证作为首要评估信号;通过提示模板配置应用特定规则;支持生产级鲁棒归一化处理,如对GROUP BY的容忍、ORDER BY默认值和LIMIT启发式。实证结果表明,STEF首次实现无需模式依赖的规模化结构化查询评估,支持持续监控与代理改进闭环。
原文摘要 · Abstract (English)
Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation methodologies whether rule-based SQL matching or schema-dependent semantic parsers assume access to ground-truth queries and structured database schema, constraints that are rarely satisfied in real-world deployments. This disconnect leaves production T2SQL agents largely unevaluated beyond developer-time testing, creating silent quality degradation with no feedback mechanism for continuous improvement. We present STEF (Schema-agnostic Text-to-SQL Evaluation Framework), a production-native evaluation system that operates exclusively on natural language inputs the user question, an enriched reformulation, and the generated SQL without requiring database schema or reference queries. STEF extracts semantic specifications from both natural language and SQL representations, performs normalized feature alignment, and produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator. Key contributions include: enriched question quality validation as a first-class evaluation signal, configurable application-specific rule injection via prompt templating, and production-robust normalization handling GROUP BY tolerance, ORDER BY defaults, and LIMIT heuristics. Empirical results demonstrate that STEF enables continuous production monitoring and agent improvement feedback loops without schema dependency, making structured query evaluation viable at scale for the first time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。