用计算特征预测训练时间，帮模型节省成本和时长。
PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time
- 结合层参数、运算量和内存使用，分析网络特性
- 在未测试硬件上预测训练时长，准确率提升72%
- 适合需要优化训练成本的研究者和工程师
训练深度学习模型，尤其是基于Transformer的大型语言模型（LLMs），需要大量计算资源和长时间训练。尽管优化配置和选择合适基础设施能显著降低开销，但需依赖预分析工具。本文提出PreNeT，一种新型预测框架，通过整合层级参数、算术操作和内存利用率等综合计算指标，实现对训练时长的精准预测。其核心优势在于可准确预测在未见过的硬件架构（包括新型加速器）上的训练时间。该框架通过深入分析各神经网络层的独特特征，提升了现有预测方法的性能。实验证明，相较于当前最先进框架，PreNeT在预测准确率上最高提升72%，帮助研究者与从业者提前确定最优配置、参数与硬件方案，以实现更高的成本效益并缩短训练周期。
原文摘要 · Abstract (English)
Training deep learning models, particularly Transformer-based architectures such as Large Language Models (LLMs), demands substantial computational resources and extended training periods. While optimal configuration and infrastructure selection can significantly reduce associated costs, this optimization requires preliminary analysis tools. This paper introduces PreNeT, a novel predictive framework designed to address this optimization challenge. PreNeT facilitates training optimization by integrating comprehensive computational metrics, including layer-specific parameters, arithmetic operations and memory utilization. A key feature of PreNeT is its capacity to accurately predict training duration on previously unexamined hardware infrastructures, including novel accelerator architectures. This framework employs a sophisticated approach to capture and analyze the distinct characteristics of various neural network layers, thereby enhancing existing prediction methodologies. Through proactive implementation of PreNeT, researchers and practitioners can determine optimal configurations, parameter settings, and hardware specifications to maximize cost-efficiency and minimize training duration. Experimental results demonstrate that PreNeT achieves up to 72% improvement in prediction accuracy compared to contemporary state-of-the-art frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。