提出量化神经网络训练效率的新框架，揭示效率随训练下降的规律。
A framework for measuring the training efficiency of a neural architecture
- 构建实验框架，通过不同停止条件衡量模型训练效率
- 发现训练效率随时间下降，且与模型规模、停止标准呈非线性关系
- 指出过拟合会干扰效率测量，适合模型评估与优化研究者参考
在神经网络系统开发中，训练效率的度量仍是一个开放问题。本文提出一种实验框架，用于衡量神经架构的训练效率。为验证该方法，我们在MNIST和CIFAR-10任务上分析了卷积神经网络（CNN）及其贝叶斯等价模型（BCNN）的训练效率。结果表明，训练效率随训练进程持续下降，且在相同模型和任务下，不同停止标准会导致效率差异。我们还发现训练停止标准、训练效率、模型规模之间存在非线性关系。此外，文章揭示了过训练可能对训练效率测量产生显著混淆效应。在不同架构间的相对效率比较中，CNN在两个数据集上均优于BCNN；更普遍地，随着学习任务复杂度提升，不同架构间训练效率的差距愈发明显。
原文摘要 · Abstract (English)
Measuring Efficiency in neural network system development is an open research problem. This paper presents an experimental framework to measure the training efficiency of a neural architecture. To demonstrate our approach, we analyze the training efficiency of Convolutional Neural Networks and Bayesian equivalents on the MNIST and CIFAR-10 tasks. Our results show that training efficiency decays as training progresses and varies across different stopping criteria for a given neural model and learning task. We also find a non-linear relationship between training stopping criteria, training Efficiency, model size, and training Efficiency. Furthermore, we illustrate the potential confounding effects of overtraining on measuring the training efficiency of a neural architecture. Regarding relative training efficiency across different architectures, our results indicate that CNNs are more efficient than BCNNs on both datasets. More generally, as a learning task becomes more complex, the relative difference in training efficiency between different architectures becomes more pronounced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。