arXiv:2510.24541cs.CL2025-10

构建了覆盖1300年历史的韩语语料库,助力语言演变研究。

Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts

  • 整合19个来源、1770万文档,覆盖6种语言与古写系统。
  • 发现1860年代韩式汉字(Idu)使用达峰值后急剧下降。
  • 适用于研究韩语演变或训练能理解古文的大型语言模型。

韩国语言史以口语与书面语差异及从汉字向韩文字母(Hangul)转变为核心特征。然而,由于缺乏可获取的历史语料库,这一语言演进在自然语言处理领域长期未受关注。为此,本文推出开放韩语历史语料库(Open Korean Historical Corpus),一个大规模、开放授权的数据集,涵盖1300年历史、6种语言以及韩式汉字(Idu)和汉谚混用等少有文献的书写系统。该语料库包含1770万份文档,共计51亿词元,来自19个来源,时间跨度从公元7世纪至2025年。利用此资源,我们对关键语言变迁进行量化分析:(1) Idu使用在1860年代达峰值后迅速衰退;(2) 从汉字向韩文的转换始于约1890年,过程迅猛;(3) 北韩词汇分化导致现代分词器的未登录词率最高可达51倍。本工作为量化历时语言分析提供了基础资源,同时可作为大语言模型的预训练语料,提升其对现代韩文中汉语词汇及古体书写系统的理解能力。

原文摘要 · Abstract (English)

The history of the Korean language is characterized by a discrepancy between its spoken and written forms and a pivotal shift from Chinese characters to the Hangul alphabet. However, this linguistic evolution has remained largely unexplored in NLP due to a lack of accessible historical corpora. To address this gap, we introduce the Open Korean Historical Corpus, a large-scale, openly licensed dataset spanning 1,300 years and 6 languages, as well as under-represented writing systems like Korean-style Sinitic (Idu) and Hanja-Hangul mixed script. This corpus contains 17.7 million documents and 5.1 billion tokens from 19 sources, ranging from the 7th century to 2025. We leverage this resource to quantitatively analyze major linguistic shifts: (1) Idu usage peaked in the 1860s before declining sharply; (2) the transition from Hanja to Hangul was a rapid transformation starting around 1890; and (3) North Korea's lexical divergence causes modern tokenizers to produce up to 51 times higher out-of-vocabulary rates. This work provides a foundational resource for quantitative diachronic analysis by capturing the history of the Korean language. Moreover, it can serve as a pre-training corpus for large language models, potentially improving their understanding of Sino-Korean vocabulary in modern Hangul as well as archaic writing systems.

历史语料韩语演化语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。