arXiv:2605.25781cs.CL2026-05

用双模型交叉验证,让人工只介入少数分歧,高效生成高精度古籍标注数据。

Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation

论文配图:Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation
图 1 · 摘自论文原文
  • 两套独立模型并行标注,一致则自动采纳,不一致交由人工审核。
  • 在13,595个字段中自动接受超85%,最终词错误率低至0.003。
  • 适合需要高精度历史文献结构化提取的研究者使用。

大规模评估历史文献中的结构化信息抽取需要高精度的真值标注,但传统人工标注成本高昂,基于大语言模型的全自动方法又易产生幻觉。本文提出双三角标注框架,一种两层人机协同机制:第一层中两个架构独立的多模态大模型并行标注文档,一致则自动采纳,不一致转入人工评审;第二层对两个系统进行交叉验证,残余冲突交由领域专家处理。该框架仅依赖模型间误差独立性假设,无需分布先验或任务特定制校准,且随模型能力提升而更自主。在1887-1906年法国医学目录《指南罗森瓦尔德》(Guides Rosenwald)上,最终词错误率为0.003。该框架自动采纳超过85%的13,595个字段。我们公开该基准数据集——首个针对罗森瓦尔德指南的结构化提取真值,以支持未来历史文献处理研究。

原文摘要 · Abstract (English)

Evaluating structured-information extraction from historical documents at scale requires high-precision ground-truth annotations, yet traditional manual labeling is expensive and fully automated pipelines built on large language models are prone to hallucination. We propose Double Triangle Annotation, a two-layer human-in-the-loop framework that leverages cross-model consensus to automate the majority of annotation work while ensuring high-precision outputs. In the first layer, two architecturally independent Multimodal Large Language Models annotate each document in parallel; when they agree, the label is auto-accepted, and disagreements are routed to a human jury. A second layer cross-checks two such systems against each other, escalating residual conflicts to a domain expert. The framework rests on a single assumption -- error independence between models -- requires no distributional priors or task-specific calibration, and becomes more autonomous as model capability improves. On the Guides Rosenwald, a corpus of French medical directories spanning 1887-1906, the framework achieves a final Word Error Rate of 0.003. Applied at scale, model consensus auto-accepts over 85% of 13,595 fields. We release the resulting benchmark -- the first structured-extraction ground truth for the Rosenwald Guides -- to support future work on historical document processing.

古籍标注人机协同高精度结构化提取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。