对比多种语言识别算法在学术数据上的表现,找出最优方案。
Sorting the Babble in Babel: Assessing the Performance of Language Identification Algorithms on the OpenAlex Database
- 用不同语料和算法测试语言识别效果,评估精度、召回率与速度。
- 标题语料+FastText在兼顾召回与效率时表现最佳。
- 结果将用于改进开放学术数据库的多语言索引,适合文献分析者参考。
本研究旨在优化开放学术数据库(OpenAlex)的语言索引,通过在人工标注的论文样本中提取的不同元数据语料上,对比多种基于Python的语言识别方法的表现。首先分析各算法、语料及语言的精确率与召回率,再评估其处理速度。随后,利用概率混淆矩阵和各语言文章频率模型,在数据库层面模拟性能表现。结果显示:若侧重精确率,使用LangID算法处理贪婪语料最优;但在召回率权重较高或需考虑处理时间时,标题语料配合FastText算法整体表现最佳。鉴于目前缺乏真正多语言的大规模书目数据库,这些结果有助于验证并推动OpenAlex在跨语言研究中的潜力。
原文摘要 · Abstract (English)
This project aims to optimize the linguistic indexing of the OpenAlex database by comparing the performance of various Python-based language identification procedures on different metadata corpora extracted from a manually-annotated article sample \footnote{OpenAlex used the results presented in this article to inform the language metadata overhaul carried out as part of its recent Walden system launch. The precision and recall performance of each algorithm, corpus, and language is first analyzed, followed by an assessment of processing speeds recorded for each algorithm and corpus type. These different performance measures are then simulated at the database level using probabilistic confusion matrices for each algorithm, corpus, and language, as well as a probabilistic modeling of relative article language frequencies for the whole OpenAlex database. Results show that procedure performance strongly depends on the importance given to each of the measures implemented: for contexts where precision is preferred, using the LangID algorithm on the greedy corpus gives the best results; however, for all cases where recall is considered at least slightly more important than precision or as soon as processing times are given any kind of consideration, the procedure that consists in the application of the FastText algorithm on the Titles corpus outperforms all other alternatives. Given the lack of truly multilingual large-scale bibliographic databases, it is hoped that these results help confirm and foster the unparalleled potential of the OpenAlex database for cross-linguistic and comprehensive measurement and evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。