构建尼泊尔语-塔芒语双语语料库，助力低资源语言机器翻译
NepTam: A Nepali-Tamang Parallel Corpus and Baseline Machine Translation Experiments
- 通过网页抓取与专家翻译，构建20K真实与80K合成双语数据集
- 基于NLLB-200微调模型实现40.92（尼泊尔语→塔芒语）的BLEU得分
- 首次为塔芒语提供高质量平行语料，适合南亚低资源语言研究者
现代机器翻译系统高度依赖高质量、大规模双语数据集，但多数南亚语言缺乏此类资源。其中，尼泊尔语和塔芒语均属低资源语言，而塔芒语更是该地区数字化程度最低的语言之一。本文填补这一空白，构建了两套双语语料库：包含20,000句的黄金标准语料NepTam20K，以及80,000句的合成语料NepTam80K，均经句级对齐并支持机器翻译任务。数据来源包括尼泊尔新闻与网络内容，经预处理、语义过滤、时态与情感平衡（仅限NepTam20K），由母语塔芒语者完成翻译，并由专业语言学家验证。语料覆盖农业、健康、教育与科技、文化及通用交流五大领域。为评估数据质量，使用mBART、M2M-100、NLLB-200及基础Transformer模型进行基线实验，其中在NLLB-200上微调取得最高得分：尼泊尔语→塔芒语40.92，塔芒语→尼泊尔语45.26。
原文摘要 · Abstract (English)
Modern Translation Systems heavily rely on high-quality, large parallel datasets for state-of-the-art performance. However, such resources are largely unavailable for most of the South Asian languages. Among them, Nepali and Tamang fall into such category, with Tamang being among the least digitally resourced languages in the region. This work addresses the gap by developing NepTam20K, a 20K gold standard parallel corpus, and NepTam80K, an 80K synthetic Nepali-Tamang parallel corpus, both sentence-aligned and designed to support machine translation. The datasets were created through a pipeline involving data scraping from Nepali news and online sources, pre-processing, semantic filtering, balancing for tense and polarity (in NepTam20K dataset), expert translation into Tamang by native speakers of the language, and verification by an expert Tamang linguist. The dataset covers five domains: Agriculture, Health, Education and Technology, Culture, and General Communication. To evaluate the dataset, baseline machine translation experiments were carried out using various multilingual pre-trained models: mBART, M2M-100, NLLB-200, and a vanilla Transformer model. The fine-tuning on the NLLB-200 achieved the highest sacreBLEU scores of 40.92 (Nepali-Tamang) and 45.26 (Tamang-Nepali).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。