小模型通过元预训练实现零样本跨语言命名实体识别
Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
- 用MAML替代部分自回归目标，让小模型快速适应新语言
- 零样本微调下F1提升2-6个百分点，收敛速度加快8%
- 对带语法标记的单字实体效果最佳，适合资源匮乏语言
低资源语言的命名实体识别通常依赖微调大型多语言模型，但在内存或延迟受限场景下不可行。本文探讨能否对小型解码器语言模型进行元预训练，使其能快速适配并零样本迁移到预训练时未见的语言。为此，我们用一阶模型无关元学习（MAML）替换部分自回归目标。以语法结构差异显著但语义相近的他加禄语和宿务语为测试集，涵盖11M至570M共四种模型规模。实验表明，在仅微调头部参数时，零样本微观F1提升2-6个百分点；全量微调后提升1-3个百分点，收敛时间最多减少8%。增益在与他加禄语格标记词si/ni共现的单字人名实体上最为明显，凸显表面锚点的重要性。
原文摘要 · Abstract (English)
Named-entity recognition (NER) in low-resource languages is usually tackled by finetuning very large multilingual LMs, an option that is often infeasible in memory- or latency-constrained settings. We ask whether small decoder LMs can be pretrained so that they adapt quickly and transfer zero-shot to languages unseen during pretraining. To this end we replace part of the autoregressive objective with first-order model-agnostic meta-learning (MAML). Tagalog and Cebuano are typologically similar yet structurally different in their actor/non-actor voice systems, and hence serve as a challenging test-bed. Across four model sizes (11 M - 570 M) MAML lifts zero-shot micro-F1 by 2-6 pp under head-only tuning and 1-3 pp after full tuning, while cutting convergence time by up to 8%. Gains are largest for single-token person entities that co-occur with Tagalog case particles si/ni, highlighting the importance of surface anchors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。