arXiv:2602.07679cs.LGcs.AI2026-02

用频谱门控提升模型表达力,不增参数却更稳定高效

Spectral Gating Networks

  • 引入频谱路径与可学习门控,动态分配高频特征能力
  • 在相同计算量下,CIFAR-10准确率达93.15%,推理快11.7倍
  • 适用于视觉、NLP、音频等多领域,兼容现有MLP结构

门控机制广泛应用,但前馈网络中如何在不牺牲稳定性与可扩展性的前提下引入高频率表达能力仍待探索。基于样条的柯尔莫哥洛夫-阿诺德网络(KAN)在高维下因网格细化导致参数膨胀和优化脆弱。为在固定参数与训练预算下注入频谱容量,我们提出频谱门控网络(SGN),一种即插即用的频谱重参数化方法。SGN在标准激活路径外增加紧凑的频谱路径,通过可学习门控使模型从稳定基线逐步分配容量至频谱特征。频谱路径采用可训练随机傅里叶特征(学习频率与相位),取代网格依赖的样条,消除分辨率依赖。混合GELU-傅里叶形式进一步提升优化鲁棒性并增强高频保真度。在视觉、NLP、音频及偏微分方程(PDE)基准上,SGN在相似计算预算下持续改进准确率-效率权衡,于CIFAR-10上达到93.15%准确率,推理速度比样条型KAN快最多11.7倍。代码与训练模型将公开。

原文摘要 · Abstract (English)

Gating mechanisms are ubiquitous, yet a complementary question in feed-forward networks remains under-explored: how to introduce frequency-rich expressivity without sacrificing stability and scalability? This tension is exposed by spline-based Kolmogorov-Arnold Network (KAN) parameterizations, where grid refinement can induce parameter growth and brittle optimization in high dimensions. To propose a stability-preserving way to inject spectral capacity into existing MLP/FFN layers under fixed parameter and training budgets, we introduce Spectral Gating Networks (SGN), a drop-in spectral reparameterization. SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior and progressively allocate capacity to spectral features during training. The spectral pathway is instantiated with trainable Random Fourier Features (learned frequencies and phases), replacing grid-based splines and removing resolution dependence. A hybrid GELU-Fourier formulation further improves optimization robustness while enhancing high-frequency fidelity. Across vision, NLP, audio, and PDE benchmarks, SGN consistently improves accuracy-efficiency trade-offs under comparable computational budgets, achieving 93.15% accuracy on CIFAR-10 and up to 11.7x faster inference than spline-based KAN variants. Code and trained models will be released.

神经网络频谱建模高效架构门控机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。