通过插入目标语言标识符，让小模型实现大模型的翻译效果。
Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation
- 在源目标间插入目标语言标识符，引导注意力聚焦于目标空间表示。
- 在EC-40上超越现有MNMT方法，MITRE-913M性能媲美商用大模型。
- 适用于需要低成本高精度多语言翻译的场景，开源可复现。
多语言神经机器翻译（MNMT）旨在实现多语言间的任意翻译。尽管专用MNMT模型训练与部署成本低，但其性能始终落后于大语言模型（LLMs）。本文提出一种新方法——注册（registering），使小型MNMT模型可媲美LLMs。具体地，在源词与目标词之间插入一组指定目标语言的人工标记（称为注册符），并通过修改注意力掩码，使目标词生成仅关注注册符激活，从而体现源词在目标语言空间中的表示。在大规模基准EC-40上的实验表明，该方法达到当前MNMT最优水平。我们进一步基于93亿句对、24种语言的公开语料预训练两个模型：MITRE（多语言翻译注册模型）。其中MITRE-913M优于NLLB-3.3B，性能接近商业级大模型，并在微调中表现出强适应性。最后，我们开源模型以促进MNMT研究：https://github.com/zhiqu22/mitre。
原文摘要 · Abstract (English)
The multilingual neural machine translation (MNMT) aims for arbitrary translations across multiple languages. Although MNMT-specific models trained on parallel data offer low costs in training and deployment, their performance consistently lags behind that of large language models (LLMs). In this work, we introduce registering, a novel method that enables a small MNMT-specific model to compete with LLMs. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method advances the state-of-the-art of MNMT. We further pre-train two models, namely MITRE (multilingual translation with registers), by 9.3 billion sentence pairs across 24 languages collected from public corpora. One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: https://github.com/zhiqu22/mitre.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。