测试小模型能否像婴儿一样从少量规则中学习，发现它不遵循人类的容忍原则。
Tolerance Principle and Small Language Model Learning
- 用小型Transformer模型在人工语法上训练，测试学习最小数据量与质量。
- 模型学习不受例外容忍度限制，与人类婴儿模式不同。
- 适合研究语言学习机制与模型泛化能力的学者参考。
现代语言模型如GPT-3、BERT和LLaMA需要大量训练数据，但在充分训练后能可靠区分合乎语法与不合语法的句子。14个月大的婴儿已能从极少数例子中学会抽象语法规则，即使存在不符合规则的例外。杨（2016）提出的容忍原则定义了规则可容忍例外数量的精确阈值，仍可被学习。本研究探讨了基于Transformer的语言模型在多小、多高质量的数据下仍能泛化规则，以检验该原则的预测。我们使用优化于小数据集的BabyBERTa（Huebner et al. 2021），在人工语法上进行训练，训练集在规模、独特句型数和规则遵循与例外比例上均有变化。结果发现，与人类婴儿不同，BabyBERTa的学习动态并不符合容忍原则。
原文摘要 · Abstract (English)
Modern language models like GPT-3, BERT, and LLaMA require massive training data, yet with sufficient training they reliably learn to distinguish grammatical from ungrammatical sentences. Children aged as young as 14 months already have the capacity to learn abstract grammar rules from very few exemplars, even in the presence of non-rule-following exceptions. Yang's (2016) Tolerance Principle defines a precise threshold for how many exceptions a rule can tolerate and still be learnable. The present study explored the minimal amount and quality of training data necessary for rules to be generalized by a transformer-based language model to test the predictions of the Tolerance Principle. We trained BabyBERTa (Huebner et al. 2021), a transformer model optimized for small datasets, on artificial grammars. The training sets varied in size, number of unique sentence types, and proportion of rule-following versus exception exemplars. We found that, unlike human infants, BabyBERTa's learning dynamics do not align with the Tolerance Principle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。