用初末权重预测模型参数,大幅降低训练开销。
Less is More: Efficient Weight Farcasting with 1-Layer Neural Network
- 仅用初始与最终权重,通过单层网络预测全程参数变化
- 在DistilBERT等模型上实现更高预测精度与更低计算量
- 适合追求高效训练的大模型研究者与工程落地场景
针对大规模深度神经网络训练中的计算挑战,本文提出一种新框架,摒弃传统优化方法,转而采用长期时间序列预测技术。该方法仅依赖初始和最终权重值,无需中间过程数据,为复杂模型架构提供轻量化参数演化预测方案。我们还设计了一种新型正则化项以提升预测性能。在合成权重序列及真实模型(如大型语言模型DistilBERT)上的实验表明,该方法在预测准确率和计算效率方面均表现更优,且额外计算开销极小,为多种任务与架构下的训练加速提供了可行路径。
原文摘要 · Abstract (English)
Addressing the computational challenges inherent in training large-scale deep neural networks remains a critical endeavor in contemporary machine learning research. While previous efforts have focused on enhancing training efficiency through techniques such as gradient descent with momentum, learning rate scheduling, and weight regularization, the demand for further innovation continues to burgeon as model sizes keep expanding. In this study, we introduce a novel framework which diverges from conventional approaches by leveraging long-term time series forecasting techniques. Our method capitalizes solely on initial and final weight values, offering a streamlined alternative for complex model architectures. We also introduce a novel regularizer that is tailored to enhance the forecasting performance of our approach. Empirical evaluations conducted on synthetic weight sequences and real-world deep learning architectures, including the prominent large language model DistilBERT, demonstrate the superiority of our method in terms of forecasting accuracy and computational efficiency. Notably, our framework showcases improved performance while requiring minimal additional computational overhead, thus presenting a promising avenue for accelerating the training process across diverse tasks and architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。