MONET框架可模拟神经网络训练，助力软硬件协同优化。
MONET: Modeling and Optimization of neural NEtwork Training from Edge to Data Centers
- 基于数据流加速器建模训练流程，支持层融合优化。
- 在ResNet-18和小规模GPT-2上验证，找到更优硬件配置。
- 用遗传算法发现激活检查点的高效权衡，适合芯片设计者。
尽管软硬件协同设计显著提升了神经网络推理效率，但训练阶段的建模仍是关键而未被充分探索的挑战。训练工作负载对内存占用和反向传播复杂度有特殊要求，现有面向推理的工具无法准确捕捉。本文提出MONET框架，用于在异构数据流加速器上建模神经网络训练过程。MONET基于经过实验验证的Stream框架，该框架能对异构数据流加速器上的神经网络推理进行建模并支持层融合。利用MONET，我们对ResNet-18和小型GPT-2进行了设计空间探索，展示了其建模训练工作流及发现更优硬件架构的能力。此外，还研究了训练中因设计空间扩大而加剧的问题，如最优层融合配置的确定。同时，借助遗传算法，我们发现了激活检查点中的有趣权衡。结果表明，面向大规模高效深度学习部署，必须采用整体化的软硬件协同设计方法。
原文摘要 · Abstract (English)
While hardware-software co-design has significantly improved the efficiency of neural network inference, modeling the training phase remains a critical yet underexplored challenge. Training workloads impose distinct constraints, particularly regarding memory footprint and backpropagation complexity, which existing inference-focused tools fail to capture. This paper introduces MONET, a framework designed to model the training of neural networks on heterogeneous dataflow accelerators. MONET builds upon Stream, an experimentally verified framework that that models the inference of neural networks on heterogeneous dataflow accelerators with layer fusion. Using MONET, we explore the design space of ResNet-18 and a small GPT-2, demonstrating the framework's capability to model training workflows and find better hardware architectures. We then further examine problems that become more complex in neural network training due to the larger design space, such as determining the best layer-fusion configuration. Additionally, we use our framework to find interesting trade-offs in activation checkpointing, with the help of a genetic algorithm. Our findings highlight the importance of a holistic approach to hardware-software co-design for scalable and efficient deep learning deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。