优化数据组织方式，让大模型训练更稳定高效。
Demystifying Data Organization for Enhanced LLM Training
- 基于样本级评分重用，设计四种数据组织准则。
- 新方法在不同规模模型上提升训练稳定性与性能。
- 适合关注训练效率与泛化能力的研究者使用。
大语言模型（LLMs）已广泛应用于多个领域，但其训练效率高度依赖有效的数据整理。尽管数据选择已受到广泛关注，但针对增强训练效果的战略性数据组织仍属研究空白，尤其因为当前大多数LLM仅训练一至数个周期。本文通过复用为数据效率预先计算的样本级评分，以极低额外开销系统性探索数据组织对训练的影响。我们提出了四个关键优化准则：边界强化、循环调度、课程连续性和局部多样性。基于此，引入两种新型数据排序方法——STR与SAW。在多种模型规模和数据量下，涵盖预训练与监督微调阶段的大量实验验证了这些准则的有效性，并证明所提方法在提升训练稳定性和性能方面的鲁棒性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. While data selection has been widely studied, the strategic data organization for enhanced training remains an underexplored area, particularly since current LLMs are often trained for only one or a few epochs. This paper systematically explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency, thereby incurring minimal additional computational overhead. We identify and formalize four key guidelines for optimizing data organization: Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity. Guided by them, we introduce two novel data ordering methods termed STR and SAW. Extensive experiments across different model scales and data sizes, encompassing both pre-training and SFT stages, validate the effectiveness of our summarized guidelines. They also demonstrate the robustness of our proposed data ordering methods in enhancing the stability and performance of LLM training. Github Link: https://github.com/microsoft/data-efficacy/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。