arXiv:2607.20537cs.LGcs.AI2026-07

让大模型学会识别表格问答结果是否可靠,只需少量标注数据即可。

ReliableTableQA:How Much Supervision Does Reliability Annotation Need?

  • 构建十类可靠性分类体系,覆盖小样本、多重比较等统计风险。
  • 仅用200个标注样本,使模型可靠性准确率提升至98%,错误回答率归零。
  • 发现强化学习微调在数据充足时无实际收益,适合高效训练场景。

我们提出ReliableTableQA框架,训练大模型判断表格问答结果的统计可靠性,而非简单回答问题是否可答。在企业分析中,即使语法正确的SQL查询也可能基于过小样本、置信区间过宽或存在严重混杂,导致结果不可靠。现有系统对此类情况仍盲目自信,我们量化其为不可靠自信回答率(UCAR)。本文贡献:(1) 构建涵盖小样本聚合、多重比较膨胀、分布尾部不匹配等十大风险的可靠性分类体系(R1-R10);(2) 设计基于上下文无关文法的程序化数据生成管道,从公开零售数据模式生成5万条带可靠性标签的训练样本,并按模式分层划分SFT/GRPO数据集;(3) 通过控制实验揭示可靠性标注所需监督量。结果表明,仅需200个分模式标注的SFT样本即可将可靠性标注F1从0.61提升至0.98,解析率从0.52升至1.00,使UCAR降为零,并在未见零售域(H&M)上实现0.997的可靠度分数。相比强基线SFT,GRPO仅在SFT不足时略有提升(100样本下精确匹配提升0.06-0.16),一旦SFT充分则无显著增益。该结果表明可靠性标注本质是数据效率问题,明确界定了强化微调的适用边界。

原文摘要 · Abstract (English)

We introduce ReliableTableQA, a framework for training an LLM to annotate the statistical reliability of tabular QA results, not whether the query is answerable, but whether the computed answer is statistically meaningful. In real enterprise analytics, a syntactically correct SQL query can return a value that is based on too small a sample, has an excessively wide confidence interval, or is too confounded to support action. Existing systems answer confidently in all such cases, a failure we quantify as the Unreliable Confident Answer Rate (UCAR). We contribute (1) a ten-category reliability taxonomy (R1-R10) covering hazards such as small-sample aggregates, multiple-comparison inflation, and distribution-tail mismatch; (2) a program-first data pipeline that generates 50,000 reliability-labeled training examples from a context-free grammar over public retail schemas, with schema-stratified SFT/GRPO splits; and (3) a controlled study of how much supervision calibrated reliability annotation actually requires. We find that a small, schema-stratified SFT set is remarkably sufficient: 200 examples raise reliability-flag F1 from 0.61 to 0.98 and parse rate from 0.52 to 1.00, drive UCAR to zero, and yield a model that generalizes to an unseen retail domain (Rel-F1 0.997 on held-out H&M). Against this strong SFT baseline, GRPO, commonly assumed to be essential, helps only when SFT is under-trained (+0.06-0.16 exact-flag-set match at 100 examples, in- and out-of-distribution) and provides no measurable benefit once SFT is adequate, a null result we confirm across a hard compound-flag slice, a strict exact-match metric, and out-of-distribution evaluation. Our findings reframe reliability annotation as a data-efficiency problem and delineate precisely when reinforcement fine-tuning does and does not pay off.

大模型表格问答可靠性评估数据效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。