用GDELT数据重建新闻全文，成本近乎为零
Free Access to World News: Reconstructing Full-Text Articles from GDELT
- 从GDELT的n-gram数据中拼接重建成完整文章
- 在2211篇美国主流媒体文章上达到95%文本相似度
- 适合做经济预测与社会科学研究的免费数据获取
新闻数据已成为多个学科的重要资源，但获取全文新闻语料仍面临高昂成本和免费替代品有限的挑战。本文提出一个名为gdeltnews的Python工具包，通过利用全球事件、语言与情绪数据库（GDELT）网络新闻n-gram 3.0数据集，以接近零成本重建报纸全文。该方法将来自全球在线新闻的重叠n-gram进行合并，还原出完整文章。我们在2211篇美国主要新闻机构的文章上进行了验证，基于Levenshtein与SequenceMatcher指标，重建文章与原始文本的相似度最高达95%。该工具为经济预测、计算社会科学、信息科学及自然语言处理提供了低成本、大规模的全文新闻数据支持。
原文摘要 · Abstract (English)
News data have become essential resources across various disciplines. Still, access to full-text news corpora remains challenging due to high costs and the limited availability of free alternatives. This paper presents a novel Python package (gdeltnews) that reconstructs full-text newspaper articles at near-zero cost by leveraging the Global Database of Events, Language, and Tone (GDELT) Web News NGrams 3.0 dataset. Our method merges overlapping n-grams extracted from global online news to rebuild complete articles. We validate the approach on a benchmark set of 2211 articles from major U.S. news outlets, achieving up to 95% text similarity against original articles based on Levenshtein and SequenceMatcher metrics. Our tool facilitates economic forecasting, computational social science, information science, and natural language processing applications by enabling free and large-scale access to full-text news data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。