用第二语言学习思路训练小模型，发现改写句子最有效
Are BabyLMs Second Language Learners?
- 从第二语言学习视角设计训练数据，强调语法规则和句式改写
- 句式改写数据使模型性能提升显著，纯改写数据效果最佳
- 词义解释数据无帮助，语法信息仅有微弱提升，适合小模型研究
本文采用语言学启发的方法应对2024年BabyLM挑战（Warstadt et al. 2023）。不同于主流第一语言学习范式，我们从第二语言（L2）学习角度出发，聚焦显性语言知识，如语法规则、词汇定义和表达方式。为此，我们使用来自Wiktionary的词义信息、由大模型生成或源自语法书的语法例句，以及句式改写数据。实验发现，词义信息（如Wiktionary）对模型表现无提升；语法信息仅带来微弱改进；最具影响力的则是句式改写数据：两个最优模型分别基于改写数据与BabyLM预训练数据混合训练，或仅用改写数据训练。该结果表明，在小模型训练中，句式多样性远比词汇解释重要。
原文摘要 · Abstract (English)
This paper describes a linguistically-motivated approach to the 2024 edition of the BabyLM Challenge (Warstadt et al. 2023). Rather than pursuing a first language learning (L1) paradigm, we approach the challenge from a second language (L2) learning perspective. In L2 learning, there is a stronger focus on learning explicit linguistic information, such as grammatical notions, definitions of words or different ways of expressing a meaning. This makes L2 learning potentially more efficient and concise. We approximate this using data from Wiktionary, grammar examples either generated by an LLM or sourced from grammar books, and paraphrase data. We find that explicit information about word meaning (in our case, Wiktionary) does not boost model performance, while grammatical information can give a small improvement. The most impactful data ingredient is sentence paraphrases, with our two best models being trained on 1) a mix of paraphrase data and data from the BabyLM pretraining dataset, and 2) exclusively paraphrase data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。