用稀疏专家架构提升小模型质量，但会显著降低推理速度。
Sparse Upcycling: Inference Inefficient Finetuning
- 将密集模型转为稀疏专家结构，提升参数量与性能
- 在部分场景下效果比持续预训练高20%以上
- 适合追求性能、不介意推理延迟的开发者
小型开源大语言模型因推理高效而被广泛使用，但进一步提升其性能仍是挑战。稀疏再利用是一种有前景的方法，可将预训练的密集模型转换为混合专家（MoE）架构，从而增加参数量和模型质量。本文对比了稀疏再利用与持续预训练（CPT）在不同模型规模、计算预算和预训练时长下的效果。实验表明，在某些场景下，稀疏再利用可使模型质量相对CPT提升超过20%。然而，这带来显著的推理开销，导致大模型在高负载推理场景下延迟增加40%。研究揭示了模型质量与推理效率之间的权衡，为希望在性能与部署约束间取得平衡的实践者提供了参考。
原文摘要 · Abstract (English)
Small, highly trained, open-source large language models are widely used due to their inference efficiency, but further improving their quality remains a challenge. Sparse upcycling is a promising approach that transforms a pretrained dense model into a Mixture-of-Experts (MoE) architecture, increasing the model's parameter count and quality. In this work, we compare the effectiveness of sparse upcycling against continued pretraining (CPT) across different model sizes, compute budgets, and pretraining durations. Our experiments show that sparse upcycling can achieve better quality, with improvements of over 20% relative to CPT in certain scenarios. However, this comes with a significant inference cost, leading to 40% slowdowns in high-demand inference settings for larger models. Our findings highlight the trade-off between model quality and inference efficiency, offering insights for practitioners seeking to balance model quality and deployment constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。