从频率视角重看门控机制，提出轻量高效的新模型GmNet
GmNet: Revisiting Gating Mechanisms From A Frequency View
- 基于卷积定理，从频率角度分析门控机制的作用
- GmNet有效缓解轻量模型的低频偏差，提升图像分类性能
- 适合关注模型效率与频率建模的视觉任务研究者
门控机制被广泛应用于解决长程依赖问题，通过自适应控制信息流保持计算效率。然而，其在神经网络中的工作原理缺乏理论分析。本文受卷积定理启发，从频率视角系统研究门控机制对训练动态的影响，探究逐元素乘积与激活函数在调控不同频率成分响应间的相互作用。基于此，提出轻量级门控机制网络GmNet，能高效利用多种频率成分的信息，显著减少现有轻量模型中的低频偏差。在图像分类任务中，GmNet在效果与效率上均表现出色。
原文摘要 · Abstract (English)
Gating mechanisms have emerged as an effective strategy integrated into model designs beyond recurrent neural networks for addressing long-range dependency problems. In a broad understanding, it provides adaptive control over the information flow while maintaining computational efficiency. However, there is a lack of theoretical analysis on how the gating mechanism works in neural networks. In this paper, inspired by the \textit{convolution theorem}, we systematically explore the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective. We investigate the interact between the element-wise product and activation functions in managing the responses to different frequency components. Leveraging these insights, we propose a Gating Mechanism Network (GmNet), a lightweight model designed to efficiently utilize the information of various frequency components. It minimizes the low-frequency bias present in existing lightweight models. GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。