通过提前停止二值神经网络的累加运算，大幅减少计算量。
Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation
- 根据累加过程预测最终符号，提前终止无意义的计算
- 在CIFAR-10上减少86.6%深层卷积的累加项，精度仅降0.37点
- 适用于资源受限场景，无需重新训练模型
二值神经网络在资源受限设备部署中极具吸引力，可实现小体积和低功耗推理。对于二值激活，点积变为符号控制的加减运算，但操作数量不变。每个神经元或输出通道仍需累加全部输入，尽管最终只保留符号，这往往造成浪费。随着累加进行，部分和常远偏离零点，其最终符号在最后一个输入前已高度可预测；此后所有贡献虽改变数值，却不影响最终激活结果。本文将此现象转化为一种后训练阶段的早期停止机制。通过分析训练数据集上的累加行为，尽早预测最终符号。无需重训模型参数。在理想权重顺序下统计操作数。在VGG11与CIFAR-10上，该方法对最深层卷积减少86.6%累加项，精度仅下降0.37点；若同时应用于三个最深卷积，则全网算术减少25%，精度下降1.36点。
原文摘要 · Abstract (English)
Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference. For binary activations, the dot products become sign-controlled additions or subtractions, but the number of operations is unchanged. Indeed, every neuron or output channel still accumulates all of its input, even though only the sign will be retained, which is often wasteful. As the accumulation progresses, the running partial sum frequently drifts so far from zero that its final sign becomes highly predictable long before the last term is reached; every contribution evaluated after that point changes the value of the sum but not the final output activation. This paper turns this observation into a post-training early-stopping mechanism. We characterize the behavior of the running accumulations on the training dataset and use this information to predict the final sign as soon as possible. No model parameter is retrained. We count the number of operations under an idealized ordering of weights. On VGG11 applied to the CIFAR-10 dataset, the method removes $86.6\%$ of the accumulation terms of the deepest convolution for a $0.37$-point accuracy drop, and $25\%$ of the full-network arithmetic when used on the three deepest convolutions simultaneously, for a $1.36$-point drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。