构建多方言孟加拉语翻译系统,直接转换标准语与11种方言
Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus

- 统一多向神经机器翻译框架,跳过标准语中间环节
- 自建1.4万行多方言语料库,模型性能超越NLLB-200达13.96点BLEU
- 适合关注语言公平、低资源方言翻译的研究者与开发者
神经机器翻译与大语言模型在跨语言任务中表现优异,但难以捕捉同一语言内部的形态变化,边缘化方言使用者。在孟加拉语中,现有框架通常以标准口语孟加拉语(SCB)为中间桥梁,易引入误差并削弱跨方言差异表达。为此,本文提出一个统一的多方向NMT系统,实现SCB与11种区域方言间的直接翻译。首先回顾既有方言NLP资源,识别技术缺口。作为基础贡献,构建并发布大规模多方言平行语料库,整合7个已有数据集并经母语者验证人工扩充,共含14,562条对齐行和51,541个非空句子对。基于该语料库,采用参数高效的权值分解低秩适应(DoRA)方法对主流序列到序列架构进行基准测试。结果表明,深度单语预训练比扩大多语言容量更有效。紧凑版BanglaT5模型在性能上超越NLLB-200和mBART-50,最高提升13.96 BLEU,达到29.26 BLEU、57.26 chrF++、49.68 METEOR。数据量研究显示,超过3,000对平行句后收益递减,且方言与标准语的语言相近性比数据量更重要。最后,将优化模型部署为INT8量化网页应用,提供可扩展的开源框架,推动包容性语言技术与数字平等访问。
原文摘要 · Abstract (English)
Neural Machine Translation (NMT) and Large Language Models (LLMs) excel at cross-lingual tasks but often fail to capture intra-lingual morphological variation, marginalizing dialectal speakers. In Bangla, existing translation frameworks commonly rely on Standard Colloquial Bangla (SCB) as an intermediate pivot, which can compound errors and reduce cross-dialectal nuance. To address this gap, we introduce a unified, multi-directional NMT system for direct translation between SCB and eleven regional variants. We first review prior dialectal NLP resources to identify existing technological gaps. As a foundational contribution, we construct and release a large multi-dialect parallel corpus for Bangla, comprising 14,562 aligned rows and 51,541 non-null sentence pairs through the integration of seven prior datasets and native-speaker-verified manual augmentation. Using this corpus, we benchmark state-of-the-art sequence-to-sequence architectures with parameter-efficient Weight-Decomposed Low-Rank Adaptation (DoRA). Results show that deep monolingual pre-training is more effective than large multilingual capacity for this task. The compact BanglaT5 model outperforms NLLB-200 and mBART-50 by up to 13.96 BLEU, achieving 29.26 BLEU, 57.26 chrF++, and 49.68 METEOR. A dataset scaling study shows diminishing returns beyond 3,000 parallel pairs and indicates that linguistic proximity to Standard Bangla is more important than raw data volume for translation quality. Finally, we deploy the optimized model as an INT8-quantized web application, providing a scalable, open-source framework for inclusive language technology and equitable digital access.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。