构建首个大规模塔吉克-波斯文转写语料库，用字符级Transformer实现高精度转写。
Character-Level Transformer for Tajik-Persian Transliteration with a Parallel Lexical Corpus
- 基于字符级Transformer模型，直接处理塔吉克文到波斯文的转写。
- 达到0.3216的字符错误率，精确匹配率达31.33%，优于规则与传统模型。
- 公开完整数据集与训练代码，适合跨文字系统NLP研究者使用。
本文针对塔吉克语（西里尔字母）到波斯语（波斯-阿拉伯字母）的自动转写问题，构建了一个经词典验证的平行语料库，包含52,152个单词和短语，来源包括印刷词典、百科资料及人工校验的在线资源。据我们所知，这是目前公开可用的最大规模塔吉克-波斯文词级转写语料库。基于该语料库，训练字符级序列到序列Transformer模型，并通过字符错误率（CER）和精确匹配准确率进行评估。模型取得CER 0.3216、精确匹配准确率0.3133的成绩，优于基于词典的规则方法和循环神经网络基线模型。采用束搜索（k=3）后，性能进一步提升至CER 0.3182、准确率0.3215。论文详细描述了数据收集与预处理流程、模型架构与实验设置，并报告了词性分析结果，显示不同词类间性能存在差异。所有预处理脚本、确定性划分的数据集分组及训练配置均已公开，以支持可复现性与对塔吉克语及相关波斯语方言的后续研究。该语料库可推动字符级转写、跨文字自然语言处理及词典学应用研究。
原文摘要 · Abstract (English)
This study addresses automatic transliteration from Tajik (Cyrillic script) to Persian (Perso-Arabic script). We present a curated, lexicographically verified parallel corpus of 52,152 Tajik--Persian words and short phrases, compiled from printed dictionaries, encyclopedic sources, and manually verified online resources. To the best of our knowledge, this is one of the largest publicly available word-level corpora for Tajik--Persian transliteration. Using this corpus, we train a character-level sequence-to-sequence Transformer model and evaluate it using Character Error Rate (CER) and exact-match accuracy. The Transformer achieves a CER of 0.3216 and an exact-match accuracy of 0.3133, outperforming both dictionary-based rule-based and recurrent neural baselines. With beam search (k=3), performance further improves to CER 0.3182 and accuracy 0.3215. We describe the data collection and preprocessing pipeline, model architecture, and experimental protocol, and report a part-of-speech analysis showing performance differences across lexical categories. All preprocessing scripts, deterministic splits into training, validation, and test sets, and training configurations are released to support reproducibility and further research on Tajik and related Persian dialects. The corpus supports research in character-level transliteration, cross-script NLP, and lexicographic applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。