研究量化神经网络在定点数下的表达能力，发现多数激活函数可完整表示所有定点函数。
On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic
- 从定点参数与运算误差出发，推导出量化网络表达任意定点函数的条件。
- 证明Sigmoid、ReLU、Mish等主流激活函数满足充分条件，能表达所有定点映射。
- 发现二值权重（±1）配合常见激活函数仍具完整表达力，适合边缘设备部署。
现有神经网络表达能力研究多基于实数参数与精确运算。本文研究在离散定点参数与含舍入误差的定点运算下，量化网络的表达能力。首先，针对定点算术与激活函数，给出了量化网络表示从定点向量到定点数的所有函数的必要条件与充分条件。进一步表明，Sigmoid、ReLU、ELU、SoftPlus、SiLU、Mish、GELU等多种常用激活函数均满足该充分条件，即使用这些激活函数的网络可表示所有定点函数。在激活函数满足一定温和条件下（如存在定点输入使输出为0），必要条件与充分条件一致，从而得到一大类激活函数的充要条件。最后证明，即使采用二值权重（{-1,1}）的量化网络，在实际激活函数下仍可表示所有定点函数。
原文摘要 · Abstract (English)
Existing works on the expressive power of neural networks typically assume real parameters and exact operations. In this work, we study the expressive power of quantized networks under discrete fixed-point parameters and inexact fixed-point operations with round-off errors. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for quantized networks to represent all fixed-point functions from fixed-point vectors to fixed-point numbers. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of representing all fixed-point functions. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function $σ$, there exists a fixed-point number $x$ such that $σ(x)=0$. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in $\{-1,1\}$ can also represent all fixed-point functions for practical activation functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。