MOGNET用在线生成权重的轻量结构，在少于2MB内存下实现更高精度。
MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights
- 用分组卷积与双1x1卷积结合，减少参数量
- 在线生成第二层卷积参数，内存降至2MB以下
- 支持低精度计算，适合嵌入式设备部署
本文提出一种紧凑模型架构MOGNET，适用于资源受限硬件。MOGNET采用由两个逐点卷积夹着一个分组卷积构成的精简卷积分解模块。为进一步压缩模型大小并降低片上内存需求，第二层逐点卷积的参数通过细胞自动机结构在线生成。此外，通过多路选择器机制与位移重缩放，可在不增加硬件复杂度的前提下实现低精度权值与激活值，并融合残差路径。为高效训练，还引入一种新的权值三值化方法，平衡量化层级。实验表明，在小于2MB的内存预算下，MOGNET在模型尺寸相当或更小时，准确率相比近期先进方法提升达1%以上。
原文摘要 · Abstract (English)
This paper presents a compact model architecture called MOGNET, compatible with a resource-limited hardware. MOGNET uses a streamlined Convolutional factorization block based on a combination of 2 point-wise (1x1) convolutions with a group-wise convolution in-between. To further limit the overall model size and reduce the on-chip required memory, the second point-wise convolution's parameters are on-line generated by a Cellular Automaton structure. In addition, MOGNET enables the use of low-precision weights and activations, by taking advantage of a Multiplexer mechanism with a proper Bitshift rescaling for integrating residual paths without increasing the hardware-related complexity. To efficiently train this model we also introduce a novel weight ternarization method favoring the balance between quantized levels. Experimental results show that given tiny memory budget (sub-2Mb), MOGNET can achieve higher accuracy with a clear gap up to 1% at a similar or even lower model size compared to recent state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。