提出自适应专家混合机制，提升字节级机器翻译的上下文理解能力。
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
- 通过注意力头作为上下文专家，动态选择并混合以适配不同语言编码规则
- 在Ted-59数据集上优于子词模型，参数更少且无需调参
- 适合多语言场景下追求低资源、高可扩展性的翻译系统开发者
字节级机器翻译系统在大规模多语言场景中展现出巨大潜力。Unicode编码将每个字符映射为特定字节，消除未知词问题，避免词汇表外风险，实现广泛的语言可扩展性。然而，字节级分词导致序列语义信息稀疏，难以理解。局部上下文建模已被证明能有效赋予词元初始语义，提升句子理解。但不同语言的编码规则差异要求上下文建模具备自适应能力。为此，我们提出上下文专家混合（MoCE）方法，自适应地选择和混合注意力头（视为上下文专家），增强上下文建模的灵活性，使模型可搜索更优的上下文组合。实验表明，该方法在无需大量超参数调优的情况下超越现有方法，在Ted-59数据集上优于子词模型，且参数更少。代码已公开于https://github.com/ictnlp/MoCE。
原文摘要 · Abstract (English)
Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages. This avoids out-of-vocabulary risk in multilingual translation and enables broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Mixture of Contextualization Experts (MoCE), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and allows models to search for better contextualization combinations. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. Our code is available at https://github.com/ictnlp/MoCE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。