提出自适应正交卷积，让高效灵活的CNN架构成为可能
An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures
- 通过自适应机制扩展正交卷积，支持步幅、空洞等复杂操作
- 在大规模模型中保持高效，参数越多效率反而越高
- 适合需要稳定梯度的场景，如对抗鲁棒性与生成模型
正交卷积层在对抗鲁棒性、归一化流、GAN和李普希茨约束模型等领域具有重要价值，因其能保持范数并确保梯度传播稳定。然而，其在大规模应用中受限于计算开销及对步幅、空洞、分组卷积和转置卷积等现代特性的支持不足。本文提出AOC（自适应正交卷积），在前人方法BCOP基础上实现可扩展改进，有效克服了正交卷积构建中的现有局限。实验表明，该方法可构建表达能力强且随规模增大愈发高效的模型。为推动研究进展，我们开源了实现工具Orthogonium（https://github.com/deel-ai/orthogonium）。
原文摘要 · Abstract (English)
Orthogonal convolutional layers are valuable components in multiple areas of machine learning, such as adversarial robustness, normalizing flows, GANs, and Lipschitz-constrained models. Their ability to preserve norms and ensure stable gradient propagation makes them valuable for a large range of problems. Despite their promise, the deployment of orthogonal convolution in large-scale applications is a significant challenge due to computational overhead and limited support for modern features like strides, dilations, group convolutions, and transposed convolutions. In this paper, we introduce AOC (Adaptative Orthogonal Convolution), a scalable method that extends a previous method (BCOP), effectively overcoming existing limitations in the construction of orthogonal convolutions. This advancement unlocks the construction of architectures that were previously considered impractical. We demonstrate through our experiments that our method produces expressive models that become increasingly efficient as they scale. To foster further advancement, we provide an open-source python package implementing this method, called Orthogonium ( https://github.com/deel-ai/orthogonium ) .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。