用双重检测机制提升大模型文档字段提取的可信度判断。
Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction
- 通过两种不同视角读取文档，捕捉互补的错误模式。
- 在发票数据集上实现99.1%高覆盖率下的准确率，误判风险降70%。
- 无需调参或重训练，可直接用于新场景的可信提取。
在金融对账、合规验证和采购自动化等高风险文档处理流程中，模型无声出错比完全不输出更危险。核心挑战不是提取准确率，而是可靠的信心评估：需逐字段判断结果是否可信，可用于自动化或需人工复核。现有方法如词元级对数概率、口语化信心评分及多样本自一致性，在实际阈值下均趋向全为正，无法区分可信与不可信结果。本文提出 ExtractConf，一种跨领域、字段无关的信心评估引擎，基于同一文档的两种结构化读取方式：字段导向的猎手（Hunter）在预设字段补全压力下提取信息，文档导向的映射器（Mapper）整体扫描并提取内容支撑的值。二者失败模式不同：猎手会虚构缺失字段，映射器则遗漏视觉不显著项。它们的分歧具有独立信息量。ExtractConf融合跨调用分歧、模型内部不确定性、OCR质量、图像清晰度与空间布局，构建无需领域规则或再训练的分类器。在55字段发票数据集DocILE上，取得0.928的ROC AUC，选择性预测风险较对数概率均值降低70%；在80%覆盖下准确率达99.1%，支持实用的人机协同流程。零样本迁移至CORD收据数据集获0.858 AUC；轻量级Lasso校准使ECE降低89%，Brier得分下降43%，证实信号跨文档域泛化有效。
原文摘要 · Abstract (English)
In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。