将残差网络训练视为最优控制问题，实现深层残差层权重自动消减。
Towards an Optimal Control Perspective of ResNet Training
- 用最优控制中的阶段代价惩罚中间状态输出，构建训练新范式。
- 实验证明多余深层残差层的权重会自然趋近于零。
- 为可解释的结构化剪枝提供理论支持，适合模型压缩研究者。
我们提出了一种适用于标准残差网络架构和通用损失函数的训练公式，将其建模为一个最优控制问题。通过在最优控制框架中引入阶段代价项，对隐藏状态的中间输出施加惩罚，实现对残差路径的动态调控。对于标准残差网络，我们通过后续跳跃连接与输出层传播状态来获取中间输出。实验表明，该训练动态使不必要的深层残差层权重趋向于消失，揭示了基于理论指导的层剪枝策略的可能性。
原文摘要 · Abstract (English)
We propose a training formulation for ResNets reflecting an optimal control problem that is applicable for standard architectures and general loss functions. We suggest bridging both worlds via penalizing intermediate outputs of hidden states corresponding to stage cost terms in optimal control. For standard ResNets, we obtain intermediate outputs by propagating the state through the subsequent skip connections and the output layer. We demonstrate that our training dynamic biases the weights of the unnecessary deeper residual layers to vanish. This indicates the potential for a theory-grounded layer pruning strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。