让逻辑门网络自动优化连接,用更少门数实现更高精度。
Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks
- 通过概率选择最优连接,同时学习门类型和连接关系。
- 两层8000门达98.92%准确率,仅需固定连接模型约1/50的门数。
- 适合追求高效率低资源的神经网络硬件部署场景。
我们提出一种新方法,可对深度可微逻辑门网络(LGNs)和查找表网络(LUTNs)进行部分或完全连接优化。训练过程中,每个门或查找表输入引脚基于连接概率分布选择最优连接,同时并行学习最优门类型或查表项。在Yin-Yang、MNIST手写数字和Fashion-MNIST数据集上,连接优化的LGNs性能优于固定连接版本,且所需逻辑门数量大幅减少。在两层各8000个门的配置下,MNIST上达到98.92%准确率;单层8000门时也达98.45%,相较固定连接模型减少近50倍门数。通过高学习率、直通估计器和修剪恒定输出门类型,训练稳定性可达十层。我们还提出一种支持反向传播的查表神经元结构,在六层深度网络中表现稳定。该模型参数量仅为固定连接方法的1/4,但准确率更高。该连接训练算法在LUTNs中同样有效,两层2000个6输入查表项时取得98.88%准确率。
原文摘要 · Abstract (English)
We introduce a novel method for both partial and full optimization of the connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs). Our training method utilizes a probability distribution over a set of connections per gate/lookup table (LUT) input pin, selecting the connection with highest merit, all whilst the optimal gate types or LUT-entries are learned in parallel. We show that the connection-optimized LGNs outperform standard fixed-connection LGNs on the Yin-Yang, MNIST Handwritten Digits and Fashion-MNIST benchmarks, while requiring only a fraction of the number of logic gates. We achieve 98.92% on the MNIST dataset with two layers of 8000 gates. With only one layer of 8000 gates, we obtain 98.45%, showing that our method requires almost 50 times fewer gates compared to fixed-connection LGNs. Training stability up to ten layers has been ensured by employing a high learning rate, straight-through estimators and trimming constant-output gate types. Additionally, we present a LUT neuron description that enables stable training with backpropagation, tested up to 6-layer deep networks. The model requires four times fewer trainable parameters and still achieves a higher accuracy compared to the fixed-connection LGN training algorithm. Our connection-training algorithm also works well for the LUTNs, achieving an accuracy of 98.88% for two layers of 2000 6-input LUTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。