针对表格问答中大模型过度自信问题，提出高效校准方法MFA。
Calibrated Confidence Estimation for Tabular Question Answering
- 利用结构化数据的多格式序列化特性设计新置信度估计方法
- 相比采样方法降低20%调用成本，校准误差降低44%-63%
- 适合需要低成本高精度置信度评估的工业级表格问答场景
大型语言模型在表格问答中的应用日益广泛，但其在结构化数据上的置信度校准研究仍不充分。本文首次系统比较了五种置信度估计方法在五个前沿大模型和两个表格问答基准上的表现。所有模型均严重过自信（平滑ECE为0.35-0.64，而文本问答报告为0.10-0.15）。自评方法（如口语化判断、P(True)）AUROC为0.42-0.76，而扰动方法（语义熵、自一致性及本文提出的多格式一致性，MFA）达到0.78-0.86。经配对自举检验（p<0.001，Holm-Bonferroni校正）与三种子实验验证（GPT-4o-mini标准差仅0.006），结果显著。所提MFA利用结构化数据特有的无损确定性序列化变体（Markdown、HTML、JSON、CSV）进行置信度估计，较采样基线降低20% API成本；在TableBench上使平均AUROC达0.80，且可与采样方法互补：MFA+自一致性组合将AUROC从0.74提升至0.82。另一贡献为结构感知重校准，相较标准后处理方法提升AUROC达10个百分点。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap tests reject the null at p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only 0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration, improves AUROC by +10 percentage points over standard post-hoc methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。