针对低资源语言，用实体词重点掩码提升多语言模型跨语言表征能力。
Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languages
- 仅对名词、动词和命名实体进行掩码，保留更多上下文信息。
- 在英-僧、英-泰、僧-泰三组低资源语言上均超越传统方法。
- 适合需要高精度跨语言理解的低资源语言任务开发者。
多语言预训练语言模型（multiPLMs）通常通过掩码语言建模（MLM）目标进行训练，广泛应用于跨语言任务如双语语料挖掘。然而，其在低资源语言（LRLs）上的表现仍不理想。为提升现有multiPLM的表征能力，可通过持续预训练（continual pre-training）实现。已有研究显示，先以MLM后以翻译语言建模（TLM）进行持续预训练可改善跨语言表征。但MLM与TLM对输入序列中所有标记一视同仁，忽视其语言属性。本文提出一种新型掩码策略——语言实体掩码（LEM），用于持续预训练阶段。与传统方法不同，LEM仅对名词、动词及命名实体等具有更高句法重要性的词类进行掩码，并限制每实体跨度内仅掩码一个词，从而保留更多上下文。我们在三个下游任务——双语语料挖掘、平行数据整理与混合语种情感分析——中，使用英-僧、英-泰、僧-泰三组低资源语言对评估该方法。实验结果表明，采用LEM持续预训练的multiPLM在所有任务中均优于MLM+TLM方法。
原文摘要 · Abstract (English)
Multilingual Pre-trained Language models (multiPLMs), trained on the Masked Language Modelling (MLM) objective are commonly being used for cross-lingual tasks such as bitext mining. However, the performance of these models is still suboptimal for low-resource languages (LRLs). To improve the language representation of a given multiPLM, it is possible to further pre-train it. This is known as continual pre-training. Previous research has shown that continual pre-training with MLM and subsequently with Translation Language Modelling (TLM) improves the cross-lingual representation of multiPLMs. However, during masking, both MLM and TLM give equal weight to all tokens in the input sequence, irrespective of the linguistic properties of the tokens. In this paper, we introduce a novel masking strategy, Linguistic Entity Masking (LEM) to be used in the continual pre-training step to further improve the cross-lingual representations of existing multiPLMs. In contrast to MLM and TLM, LEM limits masking to the linguistic entity types nouns, verbs and named entities, which hold a higher prominence in a sentence. Secondly, we limit masking to a single token within the linguistic entity span thus keeping more context, whereas, in MLM and TLM, tokens are masked randomly. We evaluate the effectiveness of LEM using three downstream tasks, namely bitext mining, parallel data curation and code-mixed sentiment analysis using three low-resource language pairs English-Sinhala, English-Tamil, and Sinhala-Tamil. Experiment results show that continually pre-training a multiPLM with LEM outperforms a multiPLM continually pre-trained with MLM+TLM for all three tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。