用自监督学习提升低资源语言形态变化生成效果
Improving Low-Resource Morphological Inflection via Self-Supervised Objectives
- 在极低数据下用自编码器做辅助任务，表现最佳
- 随着数据增多，字符掩码语言模型更有效
- 基于词素边界采样掩码，能持续提升性能
自监督目标通过利用大规模无标签数据推动了自然语言处理的进展，但许多世界语言缺乏此类资源。令人意外的是，这类方法在字符级任务中应用较少，而小规模数据对这类任务仍有潜力。本文研究了在极端低资源条件下，自监督辅助任务对形态变化（一种与语言记录高度相关的字符级任务）的有效性，训练了19种语言和13种辅助目标的编码器-解码器变换模型。当无标签数据极少时，自编码表现最优；随着数据量增加，字符掩码语言模型（CMLM）效果更佳。尽管具有更强归纳偏置的目标在直觉上影响预测，但很少优于标准CMLM。然而，基于已知词素边界采样掩码始终能提升性能，为低资源形态建模指明了有前景的方向。
原文摘要 · Abstract (English)
Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world's languages. Surprisingly, they have not been explored much for character-level tasks, where smaller amounts of data have the potential to be beneficial. We investigate the effectiveness of self-supervised auxiliary tasks for morphological inflection -- a character-level task highly relevant for language documentation -- in extremely low-resource settings, training encoder-decoder transformers for 19 languages and 13 auxiliary objectives. Autoencoding yields the best performance when unlabeled data is very limited, while character masked language modeling (CMLM) becomes more effective as data availability increases. Though objectives with stronger inductive biases influence model predictions intuitively, they rarely outperform standard CMLM. However, sampling masks based on known morpheme boundaries consistently improves performance, highlighting a promising direction for low-resource morphological modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。