无教师模型也能高效训练小数据语言模型，效果不输传统方法。
When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?
- 通过动态加权学生间互学，替代传统教师指导。
- 在小数据集上表现媲美甚至超越有教师的模型。
- 适合资源有限、追求高效预训练的研究者。
我们提交了针对BabyLM挑战的方案，旨在推动数据高效语言模型预训练的边界。方法基于深度互学习，引入多样初始化的学生模型搜索机制。针对传统方法平均对待学生模型的问题，将加权互学习建模为双层优化问题：内层通过在线蒸馏学习紧凑的学生模型，外层优化权重以实现来自多样化学生的更好知识蒸馏。该动态加权策略无需教师模型，降低计算开销。实验表明，无教师方法可达到甚至超过有教师监督的效果。
原文摘要 · Abstract (English)
We present our submission to the BabyLM challenge, aiming to push the boundaries of data-efficient language model pretraining. Our method builds upon deep mutual learning, introducing a student model search for diverse initialization. We address the limitation of treating students equally by formulating weighted mutual learning as a bi-level optimization problem. The inner loop learns compact students through online distillation, while the outer loop optimizes weights for better knowledge distillation from diverse students. This dynamic weighting strategy eliminates the need for a teacher model, reducing computational requirements. Our evaluations show that teacher-less methods can match or surpass teacher-supervised approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。