统一处理12种突厥语的开源NLP工具，支持四种文字
TurkicNLP: An NLP Toolkit for Turkic Languages
- 一套统一接口，兼容拉丁、西里尔等四类文字
- 覆盖分词到机器翻译的全流程，支持跨语言嵌入
- 自动识别文字类型，适合多语言研究与应用
突厥语族（欧亚地区超2亿人使用）的自然语言处理仍处于碎片化状态，多数语言缺乏统一工具。我们提出TurkicNLP，一个开源Python库，为四种文字体系（拉丁、西里尔、波斯-阿拉伯、古突厥鲁尼文）的突厥语言提供一致的NLP流水线。涵盖分词、形态分析、词性标注、依存句法分析、命名实体识别、双向文字转写、跨语言句子嵌入及机器翻译，并通过单一语言无关接口实现。采用模块化多后端架构，透明集成规则式有限状态转换器与神经模型，支持自动文字检测与脚本间路由，输出遵循CoNLL-U标准以确保互操作性。代码与文档托管于https://github.com/turkic-nlp/turkicnlp。
原文摘要 · Abstract (English)
Natural language processing for the Turkic language family, spoken by over 200 million people across Eurasia, remains fragmented, with most languages lacking unified tooling and resources. We present TurkicNLP, an open-source Python library providing a single, consistent NLP pipeline for Turkic languages across four script families: Latin, Cyrillic, Perso-Arabic, and Old Turkic Runic. The library covers tokenization, morphological analysis, part-of-speech tagging, dependency parsing, named entity recognition, bidirectional script transliteration, cross-lingual sentence embeddings, and machine translation through one language-agnostic API. A modular multi-backend architecture integrates rule-based finite-state transducers and neural models transparently, with automatic script detection and routing between script variants. Outputs follow the CoNLL-U standard for full interoperability and extension. Code and documentation are hosted at https://github.com/turkic-nlp/turkicnlp .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。