为儿童设计专用语言模型，提升理解力与安全性。
KidLM: Advancing Language Models for Children -- Early Insights and Future Directions
- 构建面向儿童的用户中心数据采集流程，包含儿童创作内容。
- 提出分层掩码训练目标，更好适配儿童语言特点。
- 模型更懂低年级文本，避免刻板印象，适合教育场景。
近期研究显示大语言模型在儿童教育工具中具有潜力，但仍面临语言细节、认知需求和安全标准等挑战。本文探索儿童专用语言模型的基础路径，强调高质量预训练数据的重要性。我们提出一种以用户为中心的数据收集流程，采集并验证专为儿童撰写或由儿童创作的内容。同时引入新型训练目标——分层掩码（Stratified Masking），根据儿童语言数据动态调整掩码概率，使模型更关注儿童适用词汇与概念。实验表明，该模型在理解低年级文本方面表现优异，有效避免性别与文化刻板印象，且能捕捉儿童独特偏好。此外，本文为未来儿童语言建模研究提供可操作的洞见。
原文摘要 · Abstract (English)
Recent studies highlight the potential of large language models in creating educational tools for children, yet significant challenges remain in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards. In this paper, we explore foundational steps toward the development of child-specific language models, emphasizing the necessity of high-quality pre-training data. We introduce a novel user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children. Additionally, we propose a new training objective, Stratified Masking, which dynamically adjusts masking probabilities based on our domain-specific child language data, enabling models to prioritize vocabulary and concepts more suitable for children. Experimental evaluations demonstrate that our model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children's unique preferences. Furthermore, we provide actionable insights for future research and development in child-specific language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。