小模型通过对话式指令微调可提升性能，但需注意适应性与泛化权衡。
CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
- 对比对话与问答数据，采用顺序或合并课程微调小模型
- 100M/140M参数模型在下游任务中获稳定提升，顺序课程更优
- 适合资源有限场景下优化小模型的训练策略研究者参考
本研究探讨小规模语言模型能否从指令微调中获益。比较了对话与问答类指令微调数据集，在合并或顺序课程两种方式下，使用100M和140M参数的解码器仅模型进行实验。评估涵盖微调（SuperGLUE）与零样本测试（BLiMP、EWoK、WUGs、实体追踪及心理语言学相关性）。结果表明，指令微调在微调场景中带来小而稳定的性能提升，顺序课程优于合并数据；但提升未在零样本任务中持续显现，提示交互式适应与广泛语言泛化之间存在权衡。研究揭示将人类学习策略应用于低资源模型的潜力与局限，并建议采用混合式、课程驱动的方法，在生态受限条件下增强泛化能力。
原文摘要 · Abstract (English)
This work investigates whether small-scale LMs can benefit from instruction tuning. We compare conversational and question-answering instruction tuning datasets, applied either in a merged or sequential curriculum, using decoder-only models with 100M and 140M parameters. Evaluation spans both fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and psycholinguistic correlation) settings. Results show that instruction tuning yields small but consistent gains in fine-tuning scenarios, with sequential curricula outperforming merged data; however, improvements do not consistently transfer to zero-shot tasks, suggesting a trade-off between interaction-focused adaptation and broad linguistic generalization. These results highlight both the potential and the constraints of adapting human-inspired learning strategies to low-resource LMs, and point toward hybrid, curriculum-based approaches for enhancing generalization under ecological training limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。