分离前后向温度,让离散优化更精准高效
Improving Discrete Optimisation Via Decoupled Straight-Through Estimator
- 引入前向与后向独立温度,解耦探索与梯度分布
- 在三类任务中均超越现有STE方法,性能显著提升
- 适合需要精细控制离散变量学习的深度模型研究者
直通估计器(STE)是训练含离散变量神经网络的主流方法,通过可微代理实现基于梯度的优化。然而,现有STE变体将前向传播中的随机性(控制探索与潜在空间利用)与后向传播中的梯度分散(学习信号在类别间的分配)混为一谈,二者本质不同。本文表明,将两者绑定于单一温度参数会损失大量性能提升空间。为此提出解耦直通(Decoupled ST),仅引入前向温度 $τ_f$ 与后向温度 $τ_b$ 的独立设置。这一微小改动使探索与梯度分布可独立调节。在三类不同任务(随机二值网络、分类自编码器、可微逻辑门网络)中,Decoupled ST 均持续优于 Identity STE、Softmax STE 与 Straight-Through Gumbel-Softmax。关键发现:最优 $(τ_f, τ_b)$ 配置远偏离 $τ_f = τ_b$ 对角线,证明两者的最佳解不同,单温度方法存在根本性限制。
原文摘要 · Abstract (English)
The Straight-Through Estimator (STE) is the dominant method for training neural networks with discrete variables, enabling gradient-based optimisation by routing gradients through a differentiable surrogate. However, existing STE variants conflate two fundamentally distinct concerns: forward-pass stochasticity, which controls exploration and latent space utilisation, and backward-pass gradient dispersion i.e how learning signals are distributed across categories. We show that these concerns are qualitatively different and that tying them to a single temperature parameter leaves significant performance gains untapped. We propose Decoupled Straight-Through (Decoupled ST), a minimal modification that introduces separate temperatures for the forward pass ($τ_f$) and the backward pass ($τ_b$). This simple change enables independent tuning of exploration and gradient dispersion. Across three diverse tasks (Stochastic Binary Networks, Categorical Autoencoders, and Differentiable Logic Gate Networks), Decoupled ST consistently outperforms Identity STE, Softmax STE, and Straight-Through Gumbel-Softmax. Crucially, optimal $(τ_f, τ_b)$ configurations lie far off the diagonal $τ_f = τ_b$, confirming that the two concerns do require different answers and that single-temperature methods are fundamentally constrained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。