UniBERT通过对抗训练提升多语言模型跨语言泛化能力。
UniBERT: Adversarial Training for Language-Universal Representations
- 融合掩码语言建模、对抗训练与知识蒸馏的创新训练框架。
- 在4个任务上平均相对提升7.72%，显著优于传统基线的1.17%。
- 适合需要高效多语言处理的场景，如低资源语言应用。
本文提出UniBERT，一种紧凑的多语言语言模型，采用创新训练框架，整合掩码语言建模、对抗训练与知识蒸馏。在涵盖107种语言的精心筛选维基百科语料库上预训练，旨在降低大规模模型的计算开销，同时保持在各类自然语言处理任务中的竞争力。在命名实体识别、自然语言推理、问答和语义文本相似性四项任务上的综合评估表明，结合对抗目标的多语言训练策略显著提升了跨语言泛化能力。具体而言，UniBERT模型相比传统基线平均相对提升7.72%，而后者仅达1.17%，统计分析证实该提升具有显著性（p值=0.0181）。本工作凸显了对抗训练与知识蒸馏结合在构建可扩展、鲁棒语言模型方面的优势，推动了多语言与跨语言自然语言处理领域的发展。
原文摘要 · Abstract (English)
This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks - named entity recognition, natural language inference, question answering, and semantic textual similarity - demonstrate that our multilingual training strategy enhanced by an adversarial objective significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.17%, and statistical analysis confirms the significance of these gains (p-value = 0.0181). This work highlights the benefits of combining adversarial training and knowledge distillation to build scalable and robust language models, thus advancing the field of multilingual and cross-lingual natural language processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。