动态调整连接删除率，提升神经网络鲁棒性与泛化能力。
Dynamic DropConnect: Enhancing Neural Network Robustness through Adaptive Edge Dropping Strategies
- 根据梯度大小动态分配每条连接的删除概率，无需额外参数。
- 在多个数据集上优于传统Dropout、DropConnect及经典自适应方法Standout。
- 不增加计算开销，适合追求高效稳定的模型训练者使用。
Dropout与DropConnect是通过在训练过程中以固定概率随机禁用神经元或连接来提升模型性能的经典方法。本文提出一种新策略，为网络层中每条连接分配动态的删除率，根据梯度大小自适应调整，且不引入额外学习参数。我们在合成数据和公开数据集上进行了实验，结果表明该方法在性能上超越了Dropout、DropConnect以及经典的自适应机制Standout。此外，该方法显著提升了神经网络的鲁棒性与泛化能力，同时保持计算复杂度不变。完整实现已开源，便于研究复现。
原文摘要 · Abstract (English)
Dropout and DropConnect are well-known techniques that apply a consistent drop rate to randomly deactivate neurons or edges in a neural network layer during training. This paper introduces a novel methodology that assigns dynamic drop rates to each edge within a layer, uniquely tailoring the dropping process without incorporating additional learning parameters. We perform experiments on synthetic and openly available datasets to validate the effectiveness of our approach. The results demonstrate that our method outperforms Dropout, DropConnect, and Standout, a classic mechanism known for its adaptive dropout capabilities. Furthermore, our approach improves the robustness and generalization of neural network training without increasing computational complexity. The complete implementation of our methodology is publicly accessible for research and replication purposes at https://github.com/ericabd888/Adjusting-the-drop-probability-in-DropConnect-based-on-the-magnitude-of-the-gradient/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。