用频域结构设计紧凑神经网络层,提升边缘部署的效率与可靠性。
Compact Circulant Layers with Spectral Priors
- 在频域参数化卷积核,实现低维权重空间的结构化变分推断
- 仅用少量参数即达强基线性能,且具有更紧的Lipschitz上界
- 适合资源受限场景下的贝叶斯与确定性模型,尤其适合医疗、机器人
医疗、机器人及自动驾驶等关键领域需要内存高效、具备不确定性感知能力的神经网络,适用于边缘设备等资源受限环境。本文研究紧凑的谱循环层与块循环-块循环(BCCB)层:可通过FFT对角化的循环卷积,其权重直接定义在实数FFT半轴(1D)或半平面(2D)。在频域参数化滤波器可施加简单谱结构,实现低维权重空间中的结构化变分推断,并精确计算层的谱范数,从而获得廉价的全局Lipschitz边界和基于裕度的鲁棒性诊断。通过在Hermitian支撑上放置独立复高斯分布,我们得到离散圆/环面上平稳核的谱表示,构建出精确的平稳高斯过程先验。据此定义实用的谱先验与面向实坐标、考虑Hermitian结构的低秩加对角变分后验。实验表明,谱循环/BCCB层在多种任务中表现优异:在MNIST→Fashion-MNIST上的紧凑贝叶斯网络、冻结CIFAR-10特征上的变分头、以及CIFAR-10/Tiny ImageNet上的确定性ViT投影;其性能媲美强基线,但参数量显著减少,且具有更紧的Lipschitz证书。
原文摘要 · Abstract (English)
Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。