用符合语言习得规律的课程策略，提升小模型跨语言学习效果
Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies
- 基于语言习得理论设计精细课程，按儿童语料年龄排序
- 四种语言家族实验中，课程模型超越非课程基线表现
- 适合关注小模型认知合理性与跨语言训练的研究者
课程学习是提升小型语言模型在婴儿语言挑战赛中认知合理性的常用策略，但其效果有限。本文探究能否利用语言习得理论指导更精细的课程学习策略，为四个语言类型差异大的语言家族构建按年龄排序的儿童导向语料，实现跨语言的小型语言模型与习得启发式课程。在标准小型语言模型架构上对比三种客观课程（渐进式、向内式、MMM），这些课程精确复现了习得理论预测。结果表明，精细的习得启发式课程可显著优于非课程基线，且通过精准匹配语言特定习得理论，能有效提升小型语言模型性能。
原文摘要 · Abstract (English)
Curriculum Learning has been a popular strategy to improve the cognitive plausibility of Small-Scale Language Models (SSLMs) in the BabyLM Challenge. However, it has not led to considerable improvements over non-curriculum models. We assess whether theoretical linguistic acquisition theories can be used to specify more fine-grained curriculum learning strategies, creating age-ordered corpora of Child-Directed Speech for four typologically distant language families to implement SSLMs and acquisition-inspired curricula cross-lingually. Comparing the success of three objective curricula (Growing, Inwards and MMM) that precisely replicate the predictions of acquisition theories on a standard SSLM architecture, we find fine-grained acquisition-inspired curricula can outperform non-curriculum baselines and performance benefits of curricula strategies in SSLMs can be derived by specifying fine-grained language-specific curricula that precisely replicate language acquisition theories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。