用傅里叶对角化设计低参量神经网络层,显著改善优化条件。
Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

- 构造块循环矩阵层,参数量仅为普通层的1/B
- 在MNIST上仅用2380参数达97.5%准确率,参数减3.8倍
- 理论证明优化条件数降低310倍,适合高效模型设计
通信动力学神经网络(CDNN)将通信动力学框架中的循环-谱机制用于神经网络层设计。提出CDLinear层,为块循环结构,块大小B=2l+1,参数量仅为同维稠密层的1/B。该结构使均方损失下的权重海森矩阵在离散傅里叶变换下对角化,其特征值由输入块的傅里叶谱决定。在输入预白化条件下,总体海森条件数精确为1,经验条件数上限为1 + O(sqrt(B/N))(N样本)。使用纯NumPy实现并手工推导反向传播,通过有限差分验证梯度。在8×8 MNIST基准上,三组随机种子下,B=4的CDLinear MLP以2,380参数达到97.50%±0.23%测试准确率,对比稠密基线(8,970参数,98.15%±0.47%),参数减少3.8倍,精度损失0.65%。CD-MLP平均海森条件数为1.9e4,较稠密基线的5.9e6降低约310倍。我们将CDLinear定位为结构化矩阵层的特例,主要贡献包括海森谱的闭式诊断、块乘法序列的严谨构造及显式条件分析。同时发布参考PyTorch实现,集成至DeepSeek-V3风格的专家混合(MoE)Transformer,用于未来大规模基准测试。
原文摘要 · Abstract (English)
Communication Dynamics Neural Networks (CDNNs) apply the circulant-spectral machinery of the Communication Dynamics framework to neural-network layer design. We introduce CDLinear, a block-circulant linear layer with block size B = 2l + 1 that uses 1/B the parameters of a dense layer with the same input and output dimensions. The construction gives an explicit Fourier-domain diagnostic for optimization: for mean-squared loss, the weight Hessian is diagonalized by the discrete Fourier transform, with eigenvalues determined directly by the Fourier spectrum of the input blocks. Under input pre-whitening, the population Hessian condition number is exactly 1, and the empirical condition number is bounded by 1 + O(sqrt(B/N)) for N samples. We implement CDLinear in pure NumPy with hand-derived backward passes and verify gradients by finite differences. On the 8x8 MNIST digits benchmark, across three random seeds, a CDLinear MLP with B = 4 reaches 97.50% +/- 0.23% test accuracy using 2,380 parameters, compared with 98.15% +/- 0.47% for a dense baseline using 8,970 parameters. This gives a 3.8x parameter reduction at a 0.65% accuracy cost. The CD-MLP's mean Hessian condition number is 1.9e4, about 310x smaller than the dense baseline's 5.9e6. We position CDLinear as a special case of structured matrix neural-network layers, with the main contributions being a closed-form Hessian-spectrum diagnostic, a principled discrete sequence of block multiplicities, and an explicit conditioning analysis. We also release a reference PyTorch implementation integrating CDLinear into a DeepSeek-V3-style mixture-of-experts transformer for future large-scale benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。