提出统一动态采样机制的新型算子,揭示其训练不稳定性根源。
Intriguing Properties of Dynamic Sampling Networks
- 引入泛化算子'warping',统一变形卷积等动态采样结构
- 发现前向与反向传播存在独特不对称性,影响训练稳定
- 提出基于梯度更新的损失曲面可视化新方法
深度学习中的动态采样机制在众多计算机视觉模型中展现出有效性,但其理论分析尚无统一框架。本文通过构建并分析一种新型算子‘warping’,将现有动态采样方法统一起来。该算子是动态采样的最小实现形式,便于理论分析,并可重构可变形卷积、主动卷积单元及空间变换网络等架构。基于此形式化框架,我们对输入建模为独立同分布变量和同质随机场,进行统计分析,发现模型训练中前向与反向传播存在独特不对称性。研究证实,这些机制构成一类与传统平移不变卷积算子完全不同的正交算子。结合理论与实证分析,我们确定了动态采样网络稳定训练的必要条件。此外,还研究了离散化效应的统计影响。最后,提出一种直接利用梯度更新信息的新型损失景观可视化方法,以更深入理解学习行为。
原文摘要 · Abstract (English)
Dynamic sampling mechanisms in deep learning architectures have demonstrated utility across many computer vision models, though the theoretical analysis of these structures has not yet been unified. In this paper we connect the various dynamic sampling methods by developing and analyzing a novel operator which generalizes existing methods, which we term "warping". Warping provides a minimal implementation of dynamic sampling which is amenable to analysis, and can be used to reconstruct existing architectures including deformable convolutions, active convolutional units, and spatial transformer networks. Using our formalism, we provide statistical analysis of the operator by modeling the inputs as both IID variables and homogeneous random fields. Extending this analysis, we discover a unique asymmetry between the forward and backward pass of the model training. We demonstrate that these mechanisms represent an entirely different class of orthogonal operators to the traditional translationally invariant operators defined by convolutions. With a combination of theoretical analysis and empirical investigation, we find the conditions necessary to ensure stable training of dynamic sampling networks. In addition, statistical analysis of discretization effects are studied. Finally, we introduce a novel loss landscape visualization which utilizes gradient update information directly, to better understand learning behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。