通过词元叠加技术，大幅降低大模型预训练时间。
Efficient Pre-Training with Token Superposition
- 将连续词元合并为袋状输入，用多热交叉熵训练提升吞吐。
- 在10B模型上实现等损失下训练时间缩短2.5倍。
- 无需修改架构或优化器，适配多种模型规模与结构。
大语言模型的预训练通常成本高昂且效率低下，需复杂改动才能提高数据吞吐量。本文提出词元叠加训练（TST），一种无需修改并行策略、优化器、分词器、数据或模型结构的即插即用方法。TST分为两阶段：(i) 高效叠加阶段，将多个连续词元合并为一个袋，并使用多热交叉熵（MCE）目标进行训练；(ii) 恢复阶段，回归标准训练。我们在270M和600M参数模型上广泛评估，并在3B和10B A1B混合专家模型上验证其鲁棒性。结果表明，TST在不同设置下均优于基线损失和下游任务表现；在等损失条件下，10B A1B模型上总预训练时间最多减少2.5倍。
原文摘要 · Abstract (English)
Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughput. In this work, we present Token-Superposition Training (TST), a simple drop-in method that significantly improves the data throughput per FLOPs during pre-training without modifying the parallelism, optimizer, tokenizer, data, or model architecture. TST is done in two phases: (i) A highly efficient superposition phase where we combine many contiguous tokens into one bag and train using a multi-hot cross-entropy (MCE) objective, and (ii) a recovery phase where we revert back to standard training. We extensively evaluate TST on the scale of 270M and 600M parameters and validate on 3B and a 10B A1B mixture of experts model, demonstrating that it is highly robust in different settings. Ultimately, TST consistently outperforms baseline loss and downstream evaluations, and under equal-loss settings, TST yields up to a 2.5x reduction in total pre-training time at the 10B A1B scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。