首个对比塔吉克语与波斯语音转模型的系统性研究，揭示字节级模型显著优于传统方法。
A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures
- 采用字节/字符级架构，避免子词分词对音转的干扰
- ByT5在双向音转任务中表现最佳，最高达chrF++ 87.4
- 小数据下生成式音转模型仍优于预训练多语言模型
本文首次对塔吉克语（西里尔字母）与波斯语（阿拉伯字母）之间音转任务中的现代机器学习架构进行了系统性对比分析。核心贡献在于整合多方异构来源构建并验证了唯一平行语料库，包括众包项目、词典对、《列王纪》《玛斯纳维》等古籍文本、外交文献及术语表，初始数据集含328,253句对，经分层随机抽样形成40,000对代表性子集。实验对比六类模型：规则基线、带注意力的LSTM、字符级Transformer、从零训练的G2P Transformer、预训练多语言模型（mBART、mT5+LoRA）及字节级ByT5。结果表明，ByT5在塔吉克→波斯（chrF++ 87.4）和反向（80.1）任务中全面领先；尽管数据有限，G2P Transformer仍显著优于mBART（72.3 vs. 62.2 chrF++）。使用子词分词的mT5模型完全失效（chrF++ < 18.5）。研究证明，针对该语言对的精准音转，基于字节或字符级别的架构远优于依赖子词分词的传统多语言序列到序列模型。
原文摘要 · Abstract (English)
This paper presents the first comprehensive comparative analysis of modern machine learning architectures for transliteration between Tajik (Cyrillic script) and Persian (Arabic script). A key contribution is the creation and validation of a unique parallel corpus aggregated from multiple heterogeneous sources, including crowdsourced projects, lexicographic pairs, parallel texts of "Shahnameh", diplomatic articles, texts of "Masnavi-i Ma'navi", official terminology lists, and transliterated correspondences. The initial dataset comprised 328,253 sentence pairs; a representative subset of 40,000 pairs was formed using stratified random sampling. The experiment compared six classes of models: rule-based baseline, LSTM with attention, character-level Transformer, G2P Transformer (trained from scratch), pre-trained multilingual models (mBART, mT5 with LoRA), and byte-level ByT5. Results demonstrate the overwhelming superiority of ByT5 (chrF++ 87.4 for Tajik to Farsi, 80.1 for reverse). The G2P Transformer significantly outperformed mBART (72.3 vs. 62.2 chrF++) despite limited data. Models using subword tokenization (mT5) failed completely (chrF++ less than 18.5). The findings demonstrate that for accurate transliteration of the Tajik-Farsi pair, architectures operating at the byte or character level are unequivocally more effective than traditional multilingual Seq2Seq models relying on subword tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。