arXiv:2507.01645cs.CL2025-07

研究大模型在印尼方言上的迁移能力,发现语言关联性决定效果好坏

Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settings

  • 按语言亲缘关系分组测试模型迁移效果,区分见过、部分见过和未见过的语言
  • 多语言模型在已见语言上表现最好,但对陌生方言几乎无效,而适配器方法显著提升性能
  • 模型对目标语言的先验接触比词形切分或词汇重叠更能预测迁移成功

本文通过情感分析任务,实证研究预训练语言模型在低资源印尼地方语言上的迁移能力。评估了单语印尼BERT、多语言模型mBERT和XLM-R,以及基于适配器的MAD-X方法在十种地方语言上的零样本表现。将目标语言分为三类:预训练中出现过的(seen)、语言相关但未出现的(partially seen)、完全无关的(unseen)。结果表明,多语言模型在seen语言上表现最佳,部分见过的语言上中等,未见过的语言上表现差。MAD-X显著提升性能,尤其在seen和partially seen语言上,且无需目标语言标注数据。进一步分析显示,子词碎片化和词汇重叠与预测质量弱相关,无法解释性能差异;最稳定的预测因子是模型对语言的先前接触,无论是直接还是通过相关语言。

原文摘要 · Abstract (English)

In this paper, we investigate the transferability of pre-trained language models to low-resource Indonesian local languages through the task of sentiment analysis. We evaluate both zero-shot performance and adapter-based transfer on ten local languages using models of different types: a monolingual Indonesian BERT, multilingual models such as mBERT and XLM-R, and a modular adapter-based approach called MAD-X. To better understand model behavior, we group the target languages into three categories: seen (included during pre-training), partially seen (not included but linguistically related to seen languages), and unseen (absent and unrelated in pre-training data). Our results reveal clear performance disparities across these groups: multilingual models perform best on seen languages, moderately on partially seen ones, and poorly on unseen languages. We find that MAD-X significantly improves performance, especially for seen and partially seen languages, without requiring labeled data in the target language. Additionally, we conduct a further analysis on tokenization and show that while subword fragmentation and vocabulary overlap with Indonesian correlate weakly with prediction quality, they do not fully explain the observed performance. Instead, the most consistent predictor of transfer success is the model's prior exposure to the language, either directly or through a related language.

语言迁移多语言模型零样本印尼语

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。