在极小规模预训练中，稀疏模型在活跃参数匹配下表现优于稠密模型，但总参数相同时稠密模型更优。
Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
- 采用路由专家结构替换稠密前馈层，控制活跃或总参数量进行对比
- 活跃参数匹配下稀疏模型验证损失低0.0758，总参数匹配下稠密模型低0.0180
- 适用于研究超小型模型的架构效率与参数利用策略
我们在一个类LLaMA风格的解码器训练框架下，研究了稠密模型与混合专家（MoE）Transformer在极小规模预训练中的表现。稀疏模型用Mixtral风格的路由专家替代稠密前馈块。稠密基线通过适度缩放宽度，严格匹配活跃参数或总参数预算，而分词器、数据、优化器、调度、深度、上下文长度、归一化方式和评估协议保持不变。最佳稀疏方案包含4个专家、top-2路由、Switch式负载均衡及路由器z-loss。三次种子全数据比较显示：活跃参数匹配的稠密模型达到1.6545±0.0012的最低验证损失，稀疏模型为1.5788±0.0020，总参数匹配的稠密模型为1.5608±0.0025。活跃参数匹配下稀疏模型优势为0.0758±0.0021，总参数匹配下稠密模型优势为0.0180±0.0020。训练过程中，活跃参数匹配的优势持续扩大，而总参数匹配的稠密模型优势急剧缩小。在小于2500万参数的范围内，稀疏模型在活跃参数匹配时表现更优，但在相同总存储容量下仍不及稠密模型。
原文摘要 · Abstract (English)
We study dense and mixture-of-experts (MoE) transformers in a tiny-scale pretraining regime under a shared LLaMA-style decoder training recipe. The sparse model replaces dense feed-forward blocks with Mixtral-style routed experts. Dense baselines are modestly width-resized to tightly match either active or total parameter budgets, while tokenizer, data, optimizer, schedule, depth, context length, normalization style, and evaluation protocol are held fixed. Our best sparse recipe uses four experts, top-2 routing, Switch-style load balancing, and router z-loss. In a three-seed full-data comparison, the dense active-match model reaches 1.6545 +/- 0.0012 best validation loss, the MoE reaches 1.5788 +/- 0.0020, and the dense total-match model reaches 1.5608 +/- 0.0025. This yields a matched-active gap of 0.0758 +/- 0.0021 in the MoE's favor and a matched-total gap of 0.0180 +/- 0.0020 in the dense model's favor. Across training, the matched-active advantage grows while the matched-total dense advantage narrows sharply. In this sub-25M-parameter regime, MoE therefore improves validation loss under active-parameter matching but does not surpass dense training at equal total stored capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。