证明深度神经网络可压缩到与原宽度无关的窄网络，仍保持近似功能。
Width-Independent Compressibility of Deep Neural Networks
- 提出导数匹配与逐层重加权新方法，利用输入低维特性。
- 压缩后网络宽度仅与误差预算和有效输入维数相关，与原网络宽度无关。
- 适用于理解模型压缩机制，适合研究高效神经网络设计者。
长期以来，训练良好的神经网络可被大幅压缩而不影响性能，这一现象虽重要却未被充分理解。本文证明了具有解析激活函数的深层多层感知机存在统一可压缩性定理。对于一个深度固定、宽度任意的教师网络，存在一个同深度的窄网络，能近似表示原网络的函数。该压缩后的宽度与原网络宽度无关，其量级为 $O((\log(1/\varepsilon))^{d_{in}})$，其中 $\varepsilon$ 为误差预算，$d_{in}$ 为有效输入维度。本方法基于一种新型导数匹配技术，能感知输入的低维结构，并通过逐层重加权保留输入-输出映射关系。
原文摘要 · Abstract (English)
It has long been known that well-trained neural networks can be compressed very strongly without affecting their performance, an important phenomenon that remains poorly understood. We prove a uniform compressibility theorem for deep multilayer perceptrons with analytic activations. For a deep, wide fixed teacher network, there exists a narrow (same depth) network that approximately represents the same function as the original. The reachable compressed width is strikingly independent of the original width, but is $O((\log(1/\varepsilon))^{d_{in}})$, where $\varepsilon$ is the error budget and $d_{in}$ is the effective input dimension. Our construction involves a novel derivative-matching technique which is aware of the low-dimensional input, and a layer-wise reweighting that preserves the input-output mapping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。