为历史文档OCR设计评估框架，解决大模型偏见与古文字误插问题。
Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities
- 提出针对历史文本的新型评估指标与防污染流程。
- 12个模型中Gemini和Qwen表现最优但存在过度复古化错误。
- 适合数字人文学者在古籍数字化时选型与质量把关。
数字人文研究者日益依赖大语言模型进行历史文献数字化，但缺乏适用于基于LLM的光学字符识别（OCR）的评估框架。传统指标无法捕捉时间偏差和特定时期错误，这些对历史语料库构建至关重要。本文针对18世纪俄国民间字体文本，提出一种评估方法，解决外交誊写中的数据污染风险与系统性偏差问题。引入新的量化指标：历史字符保留率（HCPR）和古字插入率（AIR），并制定污染控制与稳定性测试协议。评估12个多模态LLM发现，Gemini与Qwen模型优于传统OCR，但存在过度假古现象——将非对应时期的古体字错误插入。后处理纠错反而降低性能。该方法为数字人文工作者提供了历史语料数字化中的模型选择与质量评估指南。
原文摘要 · Abstract (English)
Digital humanities scholars increasingly use Large Language Models for historical document digitization, yet lack appropriate evaluation frameworks for LLM-based OCR. Traditional metrics fail to capture temporal biases and period-specific errors crucial for historical corpus creation. We present an evaluation methodology for LLM-based historical OCR, addressing contamination risks and systematic biases in diplomatic transcription. Using 18th-century Russian Civil font texts, we introduce novel metrics including Historical Character Preservation Rate (HCPR) and Archaic Insertion Rate (AIR), alongside protocols for contamination control and stability testing. We evaluate 12 multimodal LLMs, finding that Gemini and Qwen models outperform traditional OCR while exhibiting over-historicization: inserting archaic characters from incorrect historical periods. Post-OCR correction degrades rather than improves performance. Our methodology provides digital humanities practitioners with guidelines for model selection and quality assessment in historical corpus digitization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。