动态调整训练顺序，让模型更高效学习常识和世界知识。
A surprisal oracle for when every layer counts
- 用不确定性模型动态构建训练课程，优先处理最难的样本。
- 在常识与世界知识任务上超越官方基线，但语法推理表现一般。
- 适用于需要提升模型泛化能力的研究者，尤其关注知识推理场景。
主动课程语言建模（ACLM）是一种由学习者主导的语言模型训练方法。我们在2023年BabyLM任务中首次提出该方法，并针对2024年任务提出更新版本。ACLM通过一个不确定性的模型动态构建训练课程，将与最不确定样本相似的其他样本优先排序。新方法改进了相似性模型的动态性，并在2023年任务中最成功的模型ELC-BERT（Charpentier and Samuel, 2023）基础上运行ACLM。实验表明，尽管模型在细粒度语法推理任务上表现不佳，但在常识和世界知识任务上优于2024年官方基线。代码已开源：https://github.com/asayeed/ActiveBaby。
原文摘要 · Abstract (English)
Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。