用噪声加速逻辑门网络训练，显著减少无效单元和精度损失
Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks
- 训练时注入Gumbel噪声配合直通估计器，加快收敛
- 训练速度提升4.5倍，未使用门数降为0，离散化差距缩小98%
- 适合追求高效部署的轻量级神经网络研究者
现代神经网络在多个基准测试中表现优异，但其高计算需求和能耗限制了实际部署。逻辑门网络（LGNs）通过学习大量逻辑门实现高效图像分类，然而训练一个解决CIFAR-10问题的网络可能需要数天至数周，且近半网络单元未被使用，导致离散化差距。该差距严重影响推理性能，阻碍实际应用。本文引入带直通估计器的Gumbel噪声，在训练中显著加速收敛，提升神经元利用率，并减少离散化差距。理论分析表明，这源于隐式的海森正则化，改善了LGN的收敛性。实验显示，训练时间缩短4.5倍，离散化差距降低98%，未使用门数减少100%。
原文摘要 · Abstract (English)
Modern neural networks demonstrate state-of-the-art performance on numerous existing benchmarks; however, their high computational requirements and energy consumption prompt researchers to seek more efficient solutions for real-world deployment. Logic gate networks (LGNs) learns a large network of logic gates for efficient image classification. However, learning a network that can solve a simple problem like CIFAR-10 can take days to weeks to train. Even then, almost half of the network remains unused, causing a discretization gap. This discretization gap hinders real-world deployment of LGNs, as the performance drop between training and inference negatively impacts accuracy. We inject Gumbel noise with a straight-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap. We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of LGNs. We train networks $4.5 \times$ faster in wall-clock time, reduce the discretization gap by $98\%$, and reduce the number of unused gates by $100\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。