大模型比传统工具更准，但耗时更长，适合处理1900-1950年中文历史文本。
A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950
- 用大模型（如GPT-4o）做分词、词性标注和命名实体识别
- 大模型在所有任务上均优于传统工具，尤其擅长处理诗歌和不同时期文本
- 适合需要高精度的历史文献分析者，但需权衡计算成本
本文对比了大语言模型（LLMs）与传统自然语言处理（NLP）工具在1900至1950年间中文历史文献上的分词、词性标注（POS）和命名实体识别（NER）表现。由于汉字表意特性、缺乏自然词边界及语言演变，此类文本分析难度高。基于上海图书馆民国期刊语料库的样本数据，传统工具如Jieba和spaCy与GPT-4o、Claude 3.5及GLM系列大模型进行比较。结果表明，大模型在各项指标上均优于传统方法，尽管计算成本显著更高，凸显准确率与效率间的权衡。此外，大模型更能应对诗歌体裁及1920年前后文本的时序差异，体现出其上下文学习能力可减少对领域特定训练数据的依赖。
原文摘要 · Abstract (English)
This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER) on Chinese texts from 1900 to 1950. Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes. Using a sample dataset from the Shanghai Library Republican Journal corpus, traditional tools such as Jieba and spaCy are compared to LLMs, including GPT-4o, Claude 3.5, and the GLM series. The results show that LLMs outperform traditional methods in all metrics, albeit at considerably higher computational costs, highlighting a trade-off between accuracy and efficiency. Additionally, LLMs better handle genre-specific challenges such as poetry and temporal variations (i.e., pre-1920 versus post-1920 texts), demonstrating that their contextual learning capabilities can advance NLP approaches to historical texts by reducing the need for domain-specific training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。