改进ReLU梯度计算，让深度网络激活最大化更有效
Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks
- 前向用标准ReLU，反向用陡峭负斜率Leaky ReLU提升优化效果
- 在多个测试基准上，新方法找到的激活峰值显著高于传统方法
- 适合需要解释性AI的场景，如神经网络可视化与可解释性研究
激活最大化（AM）旨在生成能激发深度神经网络高响应的最优输入，是可解释AI的重要方法。本文发现，对于包含ReLU或Leaky ReLU的简单函数，传统AM无法生成最优输入，质疑其实际有效性及生成图像的可解释性。为此，本文提出一种新方案：在前向传播中使用标准零斜率ReLU，而在反向传播中采用大负斜率的Leaky ReLU。该方法显著提升了AM找到的最大值。所提出的ProxyGrad算法引入一个代理网络作为梯度计算工具，其损失曲面更平滑、局部极值更少。代理网络为原网络的复制品，仅在Leaky ReLU中设置不同负斜率。此外，我们证明ProxyGrad可用于训练卷积神经网络分类器，在部分测试基准上性能优于传统网络。
原文摘要 · Abstract (English)
Activation maximization (AM) strives to generate optimal input stimuli, revealing features that trigger high responses in trained deep neural networks. AM is an important method of explainable AI. We demonstrate that AM fails to produce optimal input stimuli for simple functions containing ReLUs or Leaky ReLUs, casting doubt on the practical usefulness of AM and the visual interpretation of the generated images. This paper proposes a solution based on using Leaky ReLUs with a high negative slope in the backward pass while keeping the original, usually zero, slope in the forward pass. The approach significantly increases the maxima found by AM. The resulting ProxyGrad algorithm implements a novel optimization technique for neural networks that employs a secondary network as a proxy for gradient computation. This proxy network is designed to have a simpler loss landscape with fewer local maxima than the original network. Our chosen proxy network is an identical copy of the original network, including its weights, with distinct negative slopes in the Leaky ReLUs. Moreover, we show that ProxyGrad can be used to train the weights of Convolutional Neural Networks for classification such that, on some of the tested benchmarks, they outperform traditional networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。