提出可高效计算的卷积层谱范数上界，提升模型训练稳定性。
Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers
- 用四维卷积核的张量谱范数作为雅可比矩阵谱范数的上界。
- 上界与输入分辨率无关，且在大尺寸输入下仍保持紧致性。
- 适用于需稳定训练的CNN架构，尤其适合大规模卷积网络。
控制与卷积操作相关的雅可比矩阵的谱范数已被证明能提升CNN的泛化能力、训练稳定性和鲁棒性。现有计算方法要么过度估计该范数，要么在输入和卷积核尺寸增大时性能迅速下降。本文证明，四维卷积核的张量谱范数（至多一个常数因子）可作为卷积操作相关雅可比矩阵谱范数的上界。该新上界与输入图像分辨率无关，可微且可在训练过程中高效计算。通过实验展示了该上界如何用于改进卷积架构的性能。
原文摘要 · Abstract (English)
Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。