arXiv:2411.05183cs.CVcs.LG2024-11被引 1

CNN特征分布非高斯,而是长尾且尾部相关,适合用新模型捕捉。

Why CNN Features Are not Gaussian: A Statistical Anatomy of Deep Representations

  • 发现CNN特征服从威布尔等长尾分布,非传统高斯假设。
  • 网络越深尾部越长,特征对间存在上尾依赖关系。
  • 提出新方法建模多维特征依赖,适用于图像生成与异常检测。

深度卷积神经网络(CNN)常从几何和线性代数角度分析,但其内部特征激活的统计分布仍不明确。许多应用中隐式将深度特征视为高斯分布建模密度。本文通过跨多种架构与数据集的系统研究,发现特征激活显著偏离高斯分布,更符合威布尔等长尾分布。我们引入新型离散特征函数拷贝(DCF-Copula)方法建模多维特征依赖,发现尾部长度随网络深度增加,特征对间出现上尾依赖。这些结果与中心极限定理不符,表明存在马太过程,即语义信号逐步集中在尾部。这说明CNN擅长降噪,但不适合剔除异常值。建议使用长尾上尾依赖先验替代高斯先验以准确建模深度特征密度。代码见https://github.com/dchapman-prof/DCF-Copula。

原文摘要 · Abstract (English)

Deep convolutional neural networks (CNNs) are commonly analyzed through geometric and linear-algebraic perspectives, yet the statistical distribution of their internal feature activations remains poorly understood. In many applications, deep features are implicitly treated as Gaussian when modeling densities. In this work, we empirically examine this assumption and show that it does not accurately describe the distribution of CNN feature activations. Through a systematic study across multiple architectures and datasets, we find that the feature activations deviate substantially from Gaussian and are better characterized by Weibull and related long-tailed distributions. We further introduce a novel Discretized Characteristic Function Copula (DCF-Copula) method to model multivariate feature dependencies. We find that tail-length increases with network depth and that upper-tail dependence emerges between feature pairs. These statistical findings are not consistent with the Central Limit Theorem, and are instead indicative of a Matthew process that progressively concentrates semantic signal within the tails. These statistical findings suggest that CNNs are excellent at noise reduction, yet poor at outlier removal tasks. We recommend the use of long-tailed upper-tail-dependent priors as opposed to Gaussian priors for accurately CNN deep feature density. Code available at https://github.com/dchapman-prof/DCF-Copula

CNN特征长尾分布特征建模概率建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。