调整中英文翻译的分词粒度，低资源下效果提升明显
Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance
- 源语言用高合并次数，目标语言用低合并次数，实现不对称分词
- 在5万至50万句对下，平均译文质量提升0.7到5.32点CHRF++
- 特别适合低资源语言对，六组语言实验中有10组显著胜过传统方法
现有机器翻译研究通常采用固定的对称字节对编码（BPE）分词策略，即对源语言和目标语言使用相同的合并次数（NMO）。然而，本文发现这种统一方式并不能在所有语言对和数据量下取得最优性能。通过在不同数据规模和语言对上测试BPE分词方案，我们发现采用不对称BPE——即源语言与目标语言使用不同的合并次数——能显著提升翻译效果，尤其在低资源场景（5万、10万和50万句对）中表现突出。具体而言，在英印语翻译任务中，不对称BPE分别带来5.32、4.46和0.7的CHRF++提升（均p<0.05）。该趋势在其余六个语言对（英-泰卢固语、绍纳语、挪威语、吉尔吉斯语、豪萨语、因纽特语）中也得到验证，12个系统中有10个显著优于对称BPE。结果表明，源语言使用4K至32K的高合并次数，目标语言使用0.5K至2K的低合并次数，可获得最佳翻译性能，尤其适用于低资源机器翻译。
原文摘要 · Abstract (English)
Existing Machine Translation (MT) research often suggests a single, fixed set of hyperparameters for word segmentation models, symmetric Byte Pair Encoding (BPE), which applies the same number of merge operations (NMO) to train tokenizers for both source and target languages. However, we demonstrate that this uniform approach doesn't guarantee optimal MT performance across different language pairs and data sizes. This work investigates BPE segmentation recipes across various data volumes and language pairs to evaluate MT system performance. We find that utilizing asymmetric BPE, where the source and target languages have different NMOs, significantly improves results over the symmetric approach, especially in low-resource settings (50K, 100K, and 500K sentence pairs). Specifically, asymmetric BPE yield statistically significant ($p<0.05$) average gains of 5.32, 4.46, and 0.7 CHRF++ on English-Hindi in low-resource setups (50K, 100K, and 500K sentence pairs, respectively). We validated this trend across six additional language pairs (English and Telugu, Shona, Norwegian, Kyrgyz, Hausa, and Inuktitut), observing statistically significant improvement in 10 out of 12 systems compared to symmetric BPE. Our findings indicate a high NMO for the source (4K to 32K) and a low NMO for the target (0.5K to 2K) provides optimal results, particularly benefiting low-resource MT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。