用图像转文本模型让19世纪报纸重获可读性，生成百万级高质量历史文本数据集。
Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models
- 用120亿参数图像-文本模型对8.4万页旧报纸进行高精度文字识别
- 字符错误率仅1%，比现有最佳方法低5倍，产出140万条结构化数据
- 适合历史、社会学研究者分析19世纪新闻主题与阅读难度演变
奥斯卡·王尔德曾言：‘文学与新闻的区别在于，新闻读不懂，文学没人读。’遗憾的是，19世纪英国新闻的数字化档案常因光学字符识别（OCR）质量差而难以阅读。本文针对包含8.4万页的《十九世纪期刊版》（NCSE）数据集，采用预训练的图像-文本模型Pixtral 12B进行OCR，其字符错误率中位数为1%，比其他四种方法低5倍。新版本数据集NCSE v2.0包含140万条记录、3.21亿字，涵盖四类文本和十七个主题，支持文章识别、分类与语义分析。案例展示可用于主题相似性、可读性及事件追踪研究。该数据集免费开放，助力历史与社会学研究。如今，21世纪读者可在电脑前重新体验王尔德对19世纪新闻水准的无奈与批判。
原文摘要 · Abstract (English)
Oscar Wilde said, "The difference between literature and journalism is that journalism is unreadable, and literature is not read." Unfortunately, The digitally archived journalism of Oscar Wilde's 19th century often has no or poor quality Optical Character Recognition (OCR), reducing the accessibility of these archives and making them unreadable both figuratively and literally. This paper helps address the issue by performing OCR on "The Nineteenth Century Serials Edition" (NCSE), an 84k-page collection of 19th-century English newspapers and periodicals, using Pixtral 12B, a pre-trained image-to-text language model. The OCR capability of Pixtral was compared to 4 other OCR approaches, achieving a median character error rate of 1%, 5x lower than the next best model. The resulting NCSE v2.0 dataset features improved article identification, high-quality OCR, and text classified into four types and seventeen topics. The dataset contains 1.4 million entries, and 321 million words. Example use cases demonstrate analysis of topic similarity, readability, and event tracking. NCSE v2.0 is freely available to encourage historical and sociological research. As a result, 21st-century readers can now share Oscar Wilde's disappointment with 19th-century journalistic standards, reading the unreadable from the comfort of their own computers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。