让神经网络层宽度假设为可学习，训练时自动调整以适配任务难度。
Adaptive Width Neural Networks
- 通过反向传播联合优化每层宽度与参数，无需手动调参。
- 在表格、图像、文本等多任务上，宽度随任务难易自适应变化。
- 训练后可零成本剪枝，灵活平衡性能与计算开销，适合大模型场景。
近70年来，神经网络层数宽度通常由人工设定或通过网格搜索等自动化超参数调优方法确定。本文提出一种简单易用的方法，在训练过程中学习神经网络层的无界宽度。该方法通过标准反向传播联合优化每层的宽度与参数。我们在表格、图像、文本、序列和图等多种数据领域应用该技术，展示了宽度如何随任务难度自适应调整。该宽度学习方法的一个副产物是训练后的网络可近乎零成本地截断，实现性能与计算资源间的平滑权衡。也可动态压缩网络直至性能不下降。鉴于当前基于大规模数据训练的基座模型需数十亿参数，且因训练成本过高无法进行超参数调优，本方法为宽度学习提供了可行替代方案。
原文摘要 · Abstract (English)
For almost 70 years, researchers have typically selected the width of neural networks' layers either manually or through automated hyperparameter tuning methods such as grid search and, more recently, neural architecture search. This paper challenges the status quo by introducing an easy-to-use technique to learn an unbounded width of a neural network's layer during training. The method jointly optimizes the width and the parameters of each layer via standard backpropagation. We apply the technique to a broad range of data domains such as tables, images, text, sequences, and graphs, showing how the width adapts to the task's difficulty. A by product of our width learning approach is the easy truncation of the trained network at virtually zero cost, achieving a smooth trade-off between performance and compute resources. Alternatively, one can dynamically compress the network until performances do not degrade. In light of recent foundation models trained on large datasets, requiring billions of parameters and where hyper-parameter tuning is unfeasible due to huge training costs, our approach introduces a viable alternative for width learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。