BADGER统一评估企业级生成推理中的文本转SQL与智能体行为，贴近真实业务场景。
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
- 用LLM辅助提取复杂SQL结构，支持CTE和方言差异。
- 混合执行准确率（Hybrid-EX）达87.3%平衡准确率，κ=0.717，优于6种对比框架。
- 集成多指标评估流水线，支持企业私有环境部署与自定义评测。
企业级AI系统需将自然语言转化为SQL并执行多步骤智能体推理，其评估方式应区别于学术基准。Spider和BIRD建立了执行准确率标准；G-Eval和RAGAS推动了基于大模型的评估；近期如Spider 2.0、BEAVER和BIRD-Interact开始关注企业与智能体维度。然而，尚无单一框架能将文本转SQL评估与智能体行为评估统一为面向生产环境、经由专家判断校准的流程。本文提出由Merkle开发的BADGER，一个整合文本转SQL与智能体行为评估的统一框架。其贡献包括：第一，通过LLM辅助进行SQL组件提取，扩展Spider方法以处理含大量CTE及方言特异性的SQL；第二，提出混合执行准确率（Hybrid-EX），在确定性单元级评分前利用LLM推断结构对齐，解决列别名与数值容差的脆弱性问题。在150个人工标注的工业查询上验证，Hybrid-EX达成Cohen's kappa=0.717（95%置信区间：0.600–0.822，属显著一致）与87.3%的平衡准确率，超越全部六种对比框架（Δκ：0.322–0.502，所有p≤0.001）。第三，构建企业级智能体评估套件，集成RAGAS、G-Eval与智能体基准指标，仅新增‘过度工具使用’作为创新点。BADGER全程运行于客户受控数据环境中，支持可配置的LLM评判后端，便于快速搭建客户专属评判器与指标，作为持续评估基础而非一次性质量检测。
原文摘要 · Abstract (English)
Enterprise AI systems that translate natural language into SQL queries and orchestrate multi-step agentic reasoning pipelines require evaluation approaches fundamentally different from academic benchmarks. Spider and BIRD established execution-accuracy protocols; G-Eval and RAGAS advanced LLM-based assessment; and recent work such as Spider 2.0, BEAVER, and BIRD-Interact has begun to address enterprise and agentic dimensions. No single framework unifies text-to-SQL assessment with agentic behavior evaluation into a production-grade pipeline calibrated against human expert judgment. We present BADGER, developed at Merkle, a unified evaluation framework integrating text-to-SQL assessment with agentic behavior evaluation. BADGER offers three contributions. First, LLM-assisted SQL component extraction extending Spider methodology to handle CTE-heavy, dialect-specific SQL. Second, a hybrid execution accuracy metric (Hybrid-EX) resolving column-aliasing and numeric-tolerance brittleness by using an LLM to infer structural alignments before deterministic cell-level scoring. Validated on 150 human-annotated industry queries, Hybrid-EX achieves Cohen's kappa=0.717 [95% CI: 0.600-0.822] (Substantial agreement) and 87.3% balanced accuracy, outperforming all six competing frameworks (Delta-kappa: 0.322-0.502, all p<=0.001). Third, an enterprise agentic evaluation suite assembling RAGAS, G-Eval, and agent benchmark metrics into a unified pipeline; Excess Tool Usage is the sole novel element. BADGER runs entirely within the client's governed data environment, supports configurable LLM judge backends, and enables rapid prototyping of client-specific judges and metrics, serving as a continuous evaluation backbone rather than a one-time quality gate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。