提出神经元级模块化训练方法,让大模型更易复用且体积更小。
NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
- 在训练时按神经元拆分模型,适配各类网络结构。
- 相比现有方法,模块准确率提升1.72%,大小减少58.10%。
- 适合需要高效复用的大模型开发人员使用。
随着深度神经网络(DNN)模型在现代软件系统中的广泛应用,其高昂的构建成本已成为显著挑战。模型复用虽可降低训练成本,但直接复用整个模型会带来巨大推理开销。因此,通过分解模型实现模块复用的DNN模块化技术受到关注。新兴的训练中模块化(MwT)范式将模块化融入训练过程,优于训练后模块化方法。然而,现有MwT方法仅适用于小规模卷积神经网络(CNN),在多样化的DNN及大规模模型(尤其是基于Transformer的模型)上表现不佳。为此,我们提出NeMo,一种可扩展且通用的训练中模块化方法。NeMo以神经元为基本单元,该单元普遍存在于所有DNN中,确保对Transformer等架构的适用性。设计基于对比学习的模块化训练方法,结合有效复合损失函数,实现对大规模模型的扩展性。在两个基于Transformer的模型和四个CNN模型上,于两个分类数据集上的全面实验表明,NeMo优于当前最优的MwT方法。结果展示模块分类准确率平均提升1.72%,模块大小平均减少58.10%,证明其在CNN与大规模Transformer模型上的有效性。开源项目案例研究进一步验证了其在实际场景中的潜力,为可扩展、通用的DNN模块化提供了有前景的解决方案。
原文摘要 · Abstract (English)
With the growing incorporation of deep neural network (DNN) models into modern software systems, the prohibitive construction costs have become a significant challenge. Model reuse has been widely applied to reduce training costs, but indiscriminately reusing entire models may incur significant inference overhead. Consequently, DNN modularization has gained attention, enabling module reuse by decomposing DNN models. The emerging modularizing-while-training (MwT) paradigm, which incorporates modularization into training, outperforms modularizing-after-training approaches. However, existing MwT methods focus on small-scale CNN models at the convolutional kernel level and struggle with diverse DNNs and large-scale models, particularly Transformer-based models. To address these limitations, we propose NeMo, a scalable and generalizable MwT approach. NeMo operates at the neuron level fundamental component common to all DNNs-ensuring applicability to Transformers and various architectures. We design a contrastive learning-based modular training method with an effective composite loss function, enabling scalability to large-scale models. Comprehensive experiments on two Transformer-based models and four CNN models across two classification datasets demonstrate NeMo's superiority over state-of-the-art MwT methods. Results show average gains of 1.72% in module classification accuracy and 58.10% reduction in module size, demonstrating efficacy across both CNN and large-scale Transformer-based models. A case study on open-source projects shows NeMo's potential benefits in practical scenarios, offering a promising approach for scalable and generalizable DNN modularization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。