模拟人类工作记忆发展，提升语言模型学习效率
Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition
- 训练初期限制工作记忆，后期指数式放松约束
- 语法评估显示优于无约束或静态约束方法
- 为语言关键期机制提供新证据，适合模型架构研究者
大型语言模型虽具通用语言能力，但语言习得效率远低于人类。本研究提出将人类语言关键期中工作记忆的发育特性融入语言模型训练过程。方法在训练初期对工作记忆施加约束，并随学习进程以指数方式逐步放松。目标句法评估显示，该方法显著优于无记忆约束或静态约束的传统方法。研究不仅为设计数据高效的语言模型提供了新方向，还为工作记忆发育特征作为语言关键期潜在机制提供了间接证据。
原文摘要 · Abstract (English)
Large language models possess general linguistic abilities but acquire language less efficiently than humans. This study proposes a method for integrating the developmental characteristics of working memory during the critical period, a stage when human language acquisition is particularly efficient, into the training process of language models. The proposed method introduces a mechanism that initially constrains working memory during the early stages of training and gradually relaxes this constraint in an exponential manner as learning progresses. Targeted syntactic evaluation shows that the proposed method outperforms conventional methods without memory constraints or with static memory constraints. These findings not only provide new directions for designing data-efficient language models but also offer indirect evidence supporting the role of the developmental characteristics of working memory as the underlying mechanism of the critical period in language acquisition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。