从历史报纸扫描件中提取163亿高质量文本标记,构建可复用的处理管道。
Institutional Newspapers Pipeline: Deriving billions of high quality tokens from historical newspapers
- 分步处理报纸图像,自动分割、识别并结构化每段文字内容。
- 从147万份扫描件中提取163亿字符标记,覆盖1795至1930年期间的报纸。
- 系统轻量可定制,可在普通工作站运行,适合研究历史文本与数字人文。
历史报纸是公众生活的丰富记录,但其密集、不规则且常含噪声的版式使计算访问困难。本文提出机构报纸处理管道(Institutional Newspapers Pipeline),与波士顿公共图书馆合作设计,用于从历史报纸扫描件中提取高质量、结构化的数据集。该管道模块化设计,各步骤可解释且可自定义,整体计算开销低,可在工作站级硬件上运行。每个扫描件经过多步骤处理:图像分割为无类型作物,对每段进行OCR,随后执行文本分析、类型分类、阅读顺序检测、命名实体识别、主题分类、语言识别及预计算嵌入生成。我们对波士顿公共图书馆部分馆藏运行该管道,并公开发布结果。OCR输出共包含16.3亿个o200k_base tokens,来自8310万张独立图像作物,源自1,473,635份1795至1930年间出版的公有领域报纸扫描件。本报告详述各处理步骤方法、小模型训练过程、评估结果与数据规模测量。该工作标志着向从千万级报纸扫描件中解锁高质量数据迈出重要一步。
原文摘要 · Abstract (English)
Historical newspapers are an abundant record of public life, but their dense, irregular and sometimes noisy layouts make computational access to these materials both challenging and limited. We present the Institutional Newspapers Pipeline, a modular system we jointly designed with Boston Public Library to extract high-quality, structured datasets from historical newspaper scans. It was architected so that each step remains interpretable and customizable, and so that the pipeline as a whole remains computationally frugal enough to run on workstation-level hardware. The pipeline runs each scan through a multi-step process: it segments scans into individual type-agnostic crops and performs OCR on each resulting segment before then performing text analysis, type classification, reading order detection, named entities recognition, subject classification, language detection, and pre-computed embeddings generation on every crop. We ran this pipeline against a portion of Boston Public Library's holdings and released the results as an open dataset. The optical character recognition (OCR) output represents 16.3 billion o200k_base tokens across 83.1 million individual crops, extracted from 1,473,635 public domain newspaper scans published between 1795 and 1930. This report describes our methods for each processing step, the small models we trained, as well as the evaluation results and dataset-scale measurements we collected in the process. It accompanies the release of the pipeline, models, and dataset. We position this work as a substantial step towards unlocking high-quality data from tens of millions of newspaper scans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。