提出可学习梯度补偿框架，解决二值神经网络训练中的梯度失配问题。
SURGE: Surrogate Gradient Adaptation in Binary Neural Networks
- 设计双路径梯度补偿器，用全精度辅助分支解耦梯度流。
- 在图像分类、目标检测等任务上优于现有最优方法。
- 适合需要高效二值化模型的部署场景，如边缘计算。
二值神经网络（BNNs）的训练依赖于对不可微的二值化操作（如sign函数）的梯度近似。然而，主流方法如直通估计器（STE）及其改进版本依赖人工设计，存在梯度失配和固定范围梯度截断导致的信息损失。为此，我们提出SURrogate GradiEnt Adaptation（SURGE），一种具有理论基础的可学习梯度补偿框架。SURGE通过辅助反向传播缓解梯度失配问题。具体地，我们设计了双路径梯度补偿器（DPGC），为每个二值化层构建并行的全精度辅助分支，通过输出分解实现梯度流解耦。DPGC利用全精度分支估计超出STE一阶近似的成分，实现无偏梯度估计。为进一步提升训练稳定性，我们引入基于最优缩放因子的自适应梯度缩放器（AGS），通过基于范数的缩放动态平衡跨分支梯度贡献。在图像分类、目标检测和语言理解任务上的实验表明，SURGE性能优于当前最先进方法。
原文摘要 · Abstract (English)
The training of Binary Neural Networks (BNNs) is fundamentally based on gradient approximation for non-differentiable binarization operations (e.g., sign function). However, prevailing methods including the Straight-Through Estimator (STE) and its improved variants, rely on hand-crafted designs that suffer from gradient mismatch problem and information loss induced by fixed-range gradient clipping. To address this, we propose SURrogate GradiEnt Adaptation (SURGE), a novel learnable gradient compensation framework with theoretical grounding. SURGE mitigates gradient mismatch through auxiliary backpropagation. Specifically, we design a Dual-Path Gradient Compensator (DPGC) that constructs a parallel full-precision auxiliary branch for each binarized layer, decoupling gradient flow via output decomposition during backpropagation. DPGC enables bias-reduced gradient estimation by leveraging the full-precision branch to estimate components beyond STE's first-order approximation. To further enhance training stability, we introduce an Adaptive Gradient Scaler (AGS) based on an optimal scale factor to dynamically balance inter-branch gradient contributions via norm-based scaling. Experiments on image classification, object detection, and language understanding tasks demonstrate that SURGE performs best over state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。