arXiv:2411.01074cs.LG2024-11中稿 · International Conf…被引 1

通过激活调控实现深度网络模块化,提升可复用性与训练效率。

DNN Modularization via Activation-Driven Training

  • 基于激活输出设计三重目标,引导网络自动形成模块结构。
  • 模块权重减少24倍,重叠降低37倍,训练时间节省22%。
  • 模块替换后目标类别准确率平均提升12%,适合持续迭代场景。

深度神经网络在应对需求变化时易积累技术债务并面临高昂的重训练成本。模块化有助于提升其可复用性。现有方法在训练中或训练后分解模型,但存在权重重叠大、精度损失、仅限卷积层、引入辅助掩码增加复杂度等问题。本文提出MODA,一种基于激活驱动的模块化训练方法。通过调节层输出激活,实现类内紧密、类间分离、结构紧凑三个目标,促进模型内在模块化。在三种主流DNN模型和五个不同规模数据集上的评估表明:相比现有最先进方法,MODA将训练时间减少22%;生成的模块权重最多减少24倍,权重重叠降低37倍;保持原始模型精度,无需额外微调;在模块替换场景中,目标类别平均准确率提升12%,对其他类别影响极小。

原文摘要 · Abstract (English)

Deep Neural Networks (DNNs) tend to accrue technical debt and suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity. In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and five datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 22% less training time; (2) the resultant modules generated by MODA comprise up to 24x fewer weights and 37x less weight overlap while (3) preserving the original model's accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.

模块化训练优化DNN

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。