稀疏训练让数据少时也能高效扩展大模型，突破重复数据瓶颈。
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
- 提出新缩放定律，融合活跃参数与数据重复，预测性能更准。
- 稀疏训练可延缓数据饱和，多轮训练效果更好。
- 中等稀疏（约50%）最省数据，高稀疏适合算力充足场景。
在数据无限的密集大模型中，缩放规律已被充分研究，但稀疏性如何与有限数据交互仍不明确。本文研究数据受限环境下稀疏训练，需通过多轮训练重复使用少量唯一词元。实验覆盖最大1.92B参数模型、最高93.75%稀疏度、最多2.6B唯一词元、总训练词元达41.6B（16轮），并验证了7.68B参数模型的外推表现。发现：1. 提出新缩放律，将损失建模为活跃参数数、唯一词元数、数据重复次数和稀疏度的函数，准确预测不同算力与数据预算下的性能；2. 稀疏训练可延迟数据饱和，使多轮训练更有效；3. 固定数据量下，损失最优稀疏度约为50%，而算力最优稀疏度更高且随数据规模增长。总体而言，稀疏不仅是效率工具，更是缓解数据稀缺下缩放权衡的有效机制。代码已开源：https://github.com/boqian333/sparse-dc-scaling。
原文摘要 · Abstract (English)
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。