用德语儿童语言数据训练小模型，发现句子结构分布对学习效果影响不大
Do Construction Distributions Shape Formal Language Learning In German BabyLMs?
- 在拟合儿童语言发展的德语数据上训练小型语言模型
- 不同句子结构分布下，语法和词汇学习最终准确率几乎不变
- 复杂句子利于语法学习，碎片化句子更助词汇掌握，适合语言发展研究
我们分析了德语中儿童可及/面向儿童的语言在句子层面的结构分布，对小型语言模型在词法、句法和语义能力（及其学习轨迹）的影响。这些模型基于一项新构建的符合发展规律的德语语言数据集进行训练。结果发现，尽管训练数据中结构分布差异显著，学习轨迹仍出人意料地稳健，对最终准确率影响很小，对整体学习路径几乎无影响。虽然句法学习在复杂句子中表现更好，但词法学习在碎片化句子中取得更高分数。我们认为，在发展性合理数据上训练的语言模型能为不同语言刺激是否有利于语言学习提供新视角。
原文摘要 · Abstract (English)
We analyze the influence of utterance-level construction distributions in German child-directed/child-available speech on the resulting word-level, syntactic and semantic competence (and their underlying learning trajectories) in small LMs, which we train on a novel collection of developmentally plausible language data for German. We find that trajectories are surprisingly robust for markedly different distributions of constructions in the training data, which have little effect on final accuracies and almost no effect on global learning trajectories. While syntax learning benefits from more complex utterances, word-level learning culminates in better scores with more fragmentary utterances. We argue that LMs trained on developmentally plausible data can contribute to debates on how conducive different kinds of linguistic stimuli are to language learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。