改进逻辑门网络参数化，提升训练效率与模型精度
Light Differentiable Logic Gate Networks
- 提出新型参数化方法，降低逻辑门神经元复杂度
- 训练步数减少8.5倍，反向传播速度提升1.86倍
- 模型规模缩小4倍，适合深层网络与高效推理
可微逻辑门网络（DLGNs）在推理时表现出极高的效率并保持良好精度，但梯度消失、离散化误差及高训练成本阻碍其扩展。即使采用后续工作提出的专用参数初始化方案，增加网络深度仍会损害精度。我们发现这些问题的根源在于逻辑门神经元的底层参数化方式。为此，我们提出一种新的重参数化方法，使每门输入数量的参数量对数级缩减。对于二进制输入，该方法使模型规模缩小4倍，反向传播加速达1.86倍，训练收敛所需步数减少8.5倍。此外，该方法在CIFAR-100上的精度保持稳定，甚至优于原始参数化。
原文摘要 · Abstract (English)
Differentiable logic gate networks (DLGNs) exhibit extraordinary efficiency at inference while sustaining competitive accuracy. But vanishing gradients, discretization errors, and high training cost impede scaling these networks. Even with dedicated parameter initialization schemes from subsequent works, increasing depth still harms accuracy. We show that the root cause of these issues lies in the underlying parametrization of logic gate neurons themselves. To overcome this issue, we propose a reparametrization that also shrinks the parameter size logarithmically in the number of inputs per gate. For binary inputs, this already reduces the model size by 4x, speeds up the backward pass by up to 1.86x, and converges in 8.5x fewer training steps. On top of that, we show that the accuracy on CIFAR-100 remains stable and sometimes superior to the original parametrization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。