arXiv:2506.16862cs.LGmath.OC2025-06

用最优停止理论指导神经网络深度选择,实现高效精准的早期退出。

Optimal Depth of Neural Networks

  • 将网络每层决策建模为最优停止问题,权衡精度与计算成本。
  • 证明在收益递减条件下,最优深度有限,且可推广至Transformer与连续深度模型。
  • 提出新正则项$\ ext{L}_{\rm depth}$,实测提升效率,部分场景反升准确率。

确定神经网络的最优深度是一个根本性但具挑战性的问题,通常需耗费大量资源的实验来解决。本文提出一个形式化的理论框架,将残差网络(ResNet)的前向传播重新建模为最优停止问题。我们把隐藏表示的逐层演化视为一个序列决策过程:在每一层,模型需决定是否终止计算并做出预测,或继续深入以获得更精细的表示。该设定捕捉了精度与计算成本之间的内在权衡。本文主要理论贡献是,在残差函数收益递减的合理假设下,证明了即使在无限时域设置中,期望最优停止深度仍为有限值。基于此洞察,我们提出一种新颖且实用的正则化项 $\ ext{L}_{\rm depth}$,促使网络学习适合高效早期退出的表示。通过扩展到Transformer架构,并借助自由边界问题揭示其与连续深度模型的联系,验证了该框架的普适性。在ImageNet上的实证表明,该正则项成功诱导出理论预测的行为,显著提升计算效率,且未牺牲准确率,部分情况下反而提高。

原文摘要 · Abstract (English)

Determining the optimal depth of a neural network is a fundamental yet challenging problem, typically resolved through resource-intensive experimentation. This paper introduces a formal theoretical framework to address this question by recasting the forward pass of a deep network, specifically a Residual Network (ResNet), as an optimal stopping problem. We model the layer-by-layer evolution of hidden representations as a sequential decision process where, at each layer, a choice is made between halting computation to make a prediction or continuing to a deeper layer for a potentially more refined representation. This formulation captures the intrinsic trade-off between accuracy and computational cost. Our primary theoretical contribution is a proof that, under a plausible condition of diminishing returns on the residual functions, the expected optimal stopping depth is provably finite, even in an infinite-horizon setting. We leverage this insight to propose a novel and practical regularization term, $\mathcal{L}_{\rm depth}$, that encourages the network to learn representations amenable to efficient, early exiting. We demonstrate the generality of our framework by extending it to the Transformer architecture and exploring its connection to continuous-depth models via free-boundary problems. Empirical validation on ImageNet confirms that our regularizer successfully induces the theoretically predicted behavior, leading to significant gains in computational efficiency without compromising, and in some cases improving, final model accuracy.

深度优化正则化最优停止ResNet

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。