探究语言模型在循序渐进学习下如何偏好特定语言结构
What Kind of Language is Easy to Language-Model Under Curriculum Learning?
- 用循序渐进学习模拟语言习得过程，从简单句子开始训练
- 发现语言模型的偏好倾向显著受学习顺序影响
- 适合研究语言习得机制与模型学习偏见的学者
众多已记录的语言共享相似的特征组合，形成从极罕见（如宾语-动词-主语语序）到非常普遍（如主语-宾语-动词语序）的连续谱。一个核心问题是：在何种条件下可预测这些类型学趋势？尤其是语言模型的学习偏差是否足以再现此类模式。本研究引入语言模型的学习场景这一新维度，探索其与模型归纳偏置的交互作用。作为首次研究，我们考察了循序渐进学习（CL）——一种受发展心理学启发的学习方式——的影响，即从简单句子开始而非随机输入。通过扩展已有基于语言模型的探索方法（El-Naggar等, 2025a,b），采用一种简单的CL变体，发现CL显著改变了语言模型表现出的归纳偏置。
原文摘要 · Abstract (English)
Many of the thousands of attested languages share common configurations of features, creating a spectrum from typologically very rare (e.g., object-verb-subject word order) or impossible languages to very common combinations of features (e.g., subject-object-verb word order). One central question is under what conditions such typological tendencies can be predicted, and specifically whether the learning bias of language models (LMs) is sufficient to reproduce such patterns. In this study, we add one dimensionality to such analysis -- the learning scenario for LMs -- to explore its interaction with the inductive bias of LMs. Specifically, as a first study, we examine the effect of curriculum learning (CL), as a developmentally motivated learning scenario, i.e., starting with simpler sentences rather than randomly-ordered input. We expand existing LM-based exploration (El-Naggar et al., 2025a,b) with a simple CL variant and find that CL substantially impacts the apparent inductive bias of LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。