让无法求导的神经网络也能训练,突破传统反向传播限制。
Training Non-Differentiable Networks via Optimal Transport

- 通过最优传输思想设计新优化器,仅用前向传播实现非可导网络训练。
- 在脉冲神经网络上达93.0%准确率,远超最佳基线的79.6%。
- 适合无梯度场景,如量化网络、离散路由和脉冲神经网络等。
我们针对前向传播中存在跳变(如脉冲阈值、量化层、离散路由)的损失函数进行优化,这类情况使得传统反向传播失效。有限差分法也失败:在量化网络上,99.5%的探测点对保持损失比特相同,而平滑对照组仅为1.6%。在跳跃点处,Clarke与保守平稳性未定义,任意半径h的平滑都会使梯度陡增至1/h,因此不存在趋于零的平滑半径来定义导数。固定分辨率平稳性仍可维持。PolyStep仅从前向传播中获取信息,通过旋转多面体上的探针点排序,并沿其方向的softmax加权平均步进,该方法对应于一个KL惩罚运输规划在λ=0时的端点。第二个不可能性修正了框架:当平台宽度超过探测范围时,所有重加权规则(包括softmax)均陷入冻结,唯独列约束能逃脱。在期望下,探测方向的代价加权平均恰好等于平滑损失上的梯度步,适用于任何有界可测目标,且无需探测点避开跳跃集。经过T步迭代,其收敛速率可达O(T^{-(1/2-γ)}),在内部最优探测半径下达到子空间平稳性,当损失为Lipschitz时升级为Goldstein平稳性。在六种架构与六种无梯度基线的36组对比中,PolyStep在匹配优化器步数下表现最优;在硬性脉冲神经网络上达到93.0%准确率(基线最高79.6%),在百万变量MAX-SAT上满足92.6%命题(随机基线87.5%)。若以总评估次数匹配,则仅在注意力机制和一个最大规模的MAX-SAT问题上出现反转,其余场景中硬性MoE被持平。每步计算成本为每子空间维度一次前向传播,在420万参数规模下从零开始仍保持随机水平;当梯度存在时,Adam更快更准。
原文摘要 · Abstract (English)
We optimize losses that jump: spiking thresholds, quantized layers, and discrete routing put jumps in the forward pass, where backpropagation does not apply. Finite differences fail: at a derivative-estimating radius, 99.5% of probe pairs on a quantized network leave the loss bit-identical, against 1.6% on a smooth control. At a jump, Clarke and conservative stationarity are undefined, and any radius-$h$ smoothing steepens like $1/h$, so no vanishing radius defines a derivative. Fixed-resolution stationarity survives. PolyStep attains it from forward passes alone, ranking probe points on a rotated polytope and stepping along a softmax-weighted average of their directions, the $λ=0$ endpoint of a KL-penalized transport program. A second impossibility fixes the frame: on a plateau wider than the probes reach, every rule that reweights one cost row freezes, softmax included, and only the column constraint escapes. In expectation, the cost-weighted average of the probed directions is exactly a gradient step on a smoothed loss, for every bounded measurable objective, with no probe asked to clear the jump set. Over $T$ steps it reaches subspace stationarity at rate $O(T^{-(1/2-γ)})$ up to a bias floor with an interior optimal probe radius, upgrading to Goldstein stationarity where the loss is Lipschitz. At matched optimizer steps, PolyStep leads all 36 comparisons of six architectures against six gradient-free baselines. It reaches 93.0% on hard LIF spiking networks against 79.6% for the best tuned baseline, and satisfies 92.6% of clauses on million-variable MAX-SAT against an 87.5% random floor. Matching total evaluations instead reverses argmax attention and one MAX-SAT size, and levels hard MoE. A step costs one forward pass per subspace dimension, leaving PolyStep at chance at 4.2M parameters from scratch; where gradients exist Adam is faster and more accurate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。