arXiv:2608.09254cs.AIcs.CL2026-08

7B模型靠规则门控,比32B直接提示模型更懂业务真相。

Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline

  • 用语义层和物理目录生成规则,决定回答行为并做确定性后验校验。
  • 在400个任务中业务真相率提升0.237,成本降低71.0%。
  • 适合关注真实业务准确率而非语法正确性的数据分析系统设计者。

LLM分析代理常以SQL语法准确性评估,但生产失败表现为:同一问题存在两种有效业务定义、数据仓库无法回答、字段因模式变更已弃用,以及查询执行成功却返回错误业务数值。现有执行匹配指标无法覆盖这些场景。本文提出WarehouseReliabilityBench,包含两个合成仓库的400个冻结任务,其中约一半正确答案应为澄清、拒绝或放弃。采用固定分母与预注册配对自助法,在数值出现前锁定动词。QueryProof是一个7B代理,基于语义层与物理目录构建规则,对每项输出进行确定性后执行检查。在80个合成测试任务上,该模型相比直接提示的32B基线,业务真相率提升+0.237([+0.112, +0.375]),单位正确答案成本降低71.0%;与成本匹配的少样本基线对比,精度优势仍在,但成本差异消失。对比的是系统架构而非模型规模:32B基线未使用任何辅助结构。虚假成功从0.754降至0.351,且在可回答任务中无一返回错误数值(0/24)。13个回答针对需澄清或放弃的问题。移除路由层影响甚微(0.562对0.537),说明结果不依赖升级机制。验证集调优的路由在测试集过度放弃,拟合的置信度模型也逊于原始启发式方法。更换模板族而非任务的重采样使置信区间扩大至包含零,表明方向性结论强于量化幅度。性能增益与确定性层相关,但未进行组件消融实验。

原文摘要 · Abstract (English)

LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.

智能分析业务准确规则引擎成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。