通过四象限分阶段预训练，显著提升大模型性能。
FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy
- 按困惑度高低与差异大小将数据分为四象限，分阶段训练。
- 3B模型在MMLU和CMMLU上平均提升16.8%，是随机策略的4倍。
- 适合追求高效预训练优化的模型开发者使用。
大语言模型（LLMs）在人类语言理解与生成方面取得了显著进展，预训练数据的质量与组织方式对其性能至关重要。多阶段预训练是一种有前景的方法，但现有方法常缺乏数据划分的量化标准，依赖直觉性启发。本文提出新型四象限多阶段预训练策略（FRAME），基于将预训练过程划分为四个阶段以实现损失下降四次的公认原则。该原则建立在两个关键发现之上：首先，在高困惑度（PPL）数据上训练后接低PPL数据，可使损失显著下降两次；其次，在低困惑度差异（PD）数据上训练后接高PD数据，同样能带来两次显著损失降低与性能提升。通过将数据划分为四个象限并进行策略性排序，FRAME在3B模型上于MMLU和CMMLU基准上实现了相对于随机策略16.8%的平均性能提升，有效增强大模型表现。
原文摘要 · Abstract (English)
Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performance. Multi-stage pretraining is a promising approach, but existing methods often lack quantitative criteria for data partitioning and instead rely on intuitive heuristics. In this paper, we propose the novel Four-quadRAnt Multi-stage prEtraining strategy (FRAME), guided by the established principle of organizing the pretraining process into four stages to achieve significant loss reductions four times. This principle is grounded in two key findings: first, training on high Perplexity (PPL) data followed by low PPL data, and second, training on low PPL difference (PD) data followed by high PD data, both causing the loss to drop significantly twice and performance enhancements. By partitioning data into four quadrants and strategically organizing them, FRAME achieves a remarkable 16.8% average improvement over random across MMLU and CMMLU for the 3B model, effectively boosting LLM performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。