小模型预训练中用元学习提升效率与可解释性
Learning Dynamics of Meta-Learning in Small Model Pretraining
- 结合一阶MAML与子集掩码语言模型，优化小模型预训练
- 训练速度提升1.6倍，多语言命名实体识别性能更好
- 首次揭示训练中表征先发散后压缩的可读动态
大语言模型虽强大但成本高。本文探究元学习能否让小语言模型的预训练不仅更高效，而且更具可解释性。我们融合一阶MAML与子集掩码语言模型预训练，构建了四款类LLama的解码器模型（11M-570M参数），并在多种设置和真实应用场景下评估其在基础NLP任务上的表现。相比常规训练，本方法（i）损失降至相同水平快1.6倍；（ii）在同等计算量下提升多语言通用命名实体识别（Universal NER）的F1分数；（iii）使训练动态清晰可读：网络表征先发散（“多样化”），随后收敛至更小的共享子空间（“压缩”）。这一两阶段变化体现为有效秩曲线与注意力头熵的先升后降，且可定位各层最早特化与后期重聚的位置，形成元适应的紧凑可解释信号。代码、模型检查点与WandB日志已开源。
原文摘要 · Abstract (English)
Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integrate first-order MAML with subset-masked LM pretraining, producing four LLama-style decoder-only models (11M-570M params), and evaluate it on a fundamental NLP task with many settings and real-world applications. Compared with vanilla training, our model (i) reaches the same loss up to 1.6x sooner, (ii) improves F1 on multilingual Universal NER under equal compute, and (iii) makes the training dynamics easy to read: first the network's representations fan out ("diversify") and later they collapse into a smaller, shared subspace ("compress"). This two-stage shift shows up as a rise-and-fall in both effective-rank curves and attention-head entropy. The same curves pinpoint which layers specialise earliest and which later reconverge, giving a compact, interpretable signature of meta-adaptation. Code, checkpoints and WandB logs are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。