用简化文本和复杂度排序提升小数据预训练效果
Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
- 用大模型生成的简化文本替代重复数据，优化训练
- 小模型适合从简单到复杂的训练顺序，大模型更优为交替顺序
- 适合资源有限但想提升模型泛化能力的研究者
多数语言模型预训练研究集中在大规模数据上，而数据受限场景下的优化问题仍不明确。本文聚焦预训练中的课程学习，考察文本复杂度排序与通过简化进行数据增强的效果。基于人工撰写段落与其由大模型生成的简化版本组成的平行语料库，测试四种数据调度方式：重复暴露、由低到高复杂度、由高到低复杂度、交错混合。通过微调后的样本效率以及零样本下对语言知识、实体追踪、世界知识和常识推理的表现，评估模型表征质量。结果表明，引入简化文本比重复原始数据能更好提升微调与零样本性能；小模型在由低到高复杂度顺序下表现更佳，大模型则在交错顺序中表现最优。
原文摘要 · Abstract (English)
Most studies on language model pretraining focus on large datasets, leaving open questions about optimization in data-constrained settings. In such settings, the effects of training data order and of including alternative versions of the same text remain underexplored. We address this by studying curriculum learning in pretraining, focusing on text-complexity ordering and data augmentation via simplification. We ask: (1) Does simplifying texts enhance representation quality more than reusing the original data? and (2) Does ordering data by text complexity yield better representations? To answer, we build on a pair of parallel corpora where human-written paragraphs are aligned with LLM-simplified variants, and test four data schedules: repeated exposure, low-to-high complexity, high-to-low, and interleaved. We analyze models' representation quality from a sample efficiency perspective via fine-tuning, as well as its zero-shot performance on linguistic knowledge, entity tracking, world knowledge, and commonsense reasoning. Our findings show that adding simplified data improves fine-tuning and zero-shot performance over a repeated-exposure baseline: smaller models benefit from low-to-high complexity, while larger models perform better with interleaved ordering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。