用大规模多语言翻译模型的编码器做词对齐,效果比现有方法更好。
TransAlign: Machine Translation Encoders are Strong Word Aligners, Too
- 利用多语言翻译模型编码器提取词对齐信息。
- 在跨语言命名实体识别任务中对齐准确率超90%。
- 适合低资源语言的文本分类迁移任务。
由于大多数世界语言和自然语言处理任务缺乏足够训练数据,基于翻译的策略如 translate-test(在目标语言的噪声源语言数据上评估)和 translate-train(在从源语言翻译的目标语言噪声数据上训练)已成为跨语言迁移(XLT)的有力方法。对于词元分类任务,这些策略需要标签投影:将原句每个词的标签映射到其在翻译中的对应词。通常借助由mBERT或LaBSE等编码器模型导出的多语言词对齐器(WAs)。尽管机器翻译(MT)与词对齐存在明显关联,但现有研究大多仅利用编码器-解码器架构中的交叉注意力,导致对齐性能较差。本文提出TransAlign,一种新词对齐方法,充分利用大规模多语言翻译模型的编码器。实验表明,TransAlign不仅实现优异的词对齐性能,且在基于翻译的跨语言词元分类迁移任务中显著优于主流词对齐器及最先进的非对齐标签投影方法。
原文摘要 · Abstract (English)
In the absence of sizable training data for most world languages and NLP tasks, translation-based strategies such as translate-test -- evaluating on noisy source language data translated from the target language -- and translate-train -- training on noisy target language data translated from the source language -- have been established as competitive approaches for cross-lingual transfer (XLT). For token classification tasks, these strategies require label projection: mapping the labels from each token in the original sentence to its counterpart(s) in the translation. To this end, it is common to leverage multilingual word aligners (WAs) derived from encoder language models such as mBERT or LaBSE. Despite obvious associations between machine translation (MT) and WA, research on extracting alignments with MT models is largely limited to exploiting cross-attention in encoder-decoder architectures, yielding poor WA results. In this work, in contrast, we propose TransAlign, a novel word aligner that utilizes the encoder of a massively multilingual MT model. We show that TransAlign not only achieves strong WA performance but substantially outperforms popular WA and state-of-the-art non-WA-based label projection methods in MT-based XLT for token classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。