提出CAGE方法解决神经网络训练与推理的逻辑门选择差异问题。
Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks
- 用置信度自适应梯度估计，实现前向一致与梯度流动。
- 在MNIST上达98%准确率，CIFAR-10上超58%，且全温度下无选择偏差。
- 验证了仅靠随机噪声无法缩小训练-推理差距，适合高效架构设计者。
在神经网络中，训练时常用固定候选组件（如逻辑门）的软混合以实现稳定优化，而推理时则采用硬选择，这导致训练-推理不匹配。本文通过将前向计算（硬选择与软混合）与随机性（有无Gumbel噪声）分离，分析该差距。以逻辑门网络为测试平台，发现四种方法表现各异：Hard-ST因构造特性实现零选择差距；Gumbel-ST在训练成功时接近零差距，但低温下出现准确率崩溃；Soft-Mix仅在低温下通过权重集中实现小差距；Soft-Gumbel虽有Gumbel噪声仍存在大差距，证明噪声本身不足以缩小差距。为此提出CAGE（置信度自适应梯度估计），在保持前向对齐的同时维持梯度流动。在逻辑门网络上，结合CAGE的Hard-ST在MNIST上准确率超98%，在CIFAR-10上超58%，且所有温度下选择差距为零；而未使用CAGE的Gumbel-ST准确率下降47点。
原文摘要 · Abstract (English)
In neural network models, soft mixtures of fixed candidate components (e.g., logic gates and sub-networks) are often used during training for stable optimization, while hard selection is typically used at inference. This raises questions about training-inference mismatch. We analyze this gap by separating forward-pass computation (hard selection vs. soft mixture) from stochasticity (with vs. without Gumbel noise). Using logic gate networks as a testbed, we observe distinct behaviors across four methods: Hard-ST achieves zero selection gap by construction; Gumbel-ST achieves near-zero gap when training succeeds but suffers accuracy collapse at low temperatures; Soft-Mix achieves small gap only at low temperature via weight concentration; and Soft-Gumbel exhibits large gaps despite Gumbel noise, confirming that noise alone does not reduce the gap. We propose CAGE (Confidence-Adaptive Gradient Estimation) to maintain gradient flow while preserving forward alignment. On logic gate networks, Hard-ST with CAGE achieves over 98% accuracy on MNIST and over 58% on CIFAR-10, both with zero selection gap across all temperatures, while Gumbel-ST without CAGE suffers a 47-point accuracy collapse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。