Y-Drop让重要神经元更易被丢弃，提升模型鲁棒性。
Y-Drop: A Conductance based Dropout for fully connected layers
- 基于神经元导通率设计新丢弃策略，优先淘汰重要节点。
- 在多个数据集上优于传统Dropout，且随网络增大效果更显著。
- 适合追求高鲁棒性的深度网络训练，尤其大模型场景。
本文提出Y-Drop，一种新型正则化方法，通过神经元导通率（neuron conductance）衡量神经元重要性，并引导丢弃算法更倾向于淘汰重要神经元。该导通率是可解释的指标，反映每个神经元对端到端映射的贡献。实验表明，强制网络在缺少关键神经元的情况下完成任务，能产生强正则化效应。进一步分析显示，使用Y-Drop后，更多神经元具有高导通率，模型更具鲁棒性。在多个数据集和网络结构组合上，Y-Drop的正则化效果随模型规模增长优于标准Dropout，且几乎无需调参即可持续取得更优性能。
原文摘要 · Abstract (English)
In this work, we introduce Y-Drop, a regularization method that biases the dropout algorithm towards dropping more important neurons with higher probability. The backbone of our approach is neuron conductance, an interpretable measure of neuron importance that calculates the contribution of each neuron towards the end-to-end mapping of the network. We investigate the impact of the uniform dropout selection criterion on performance by assigning higher dropout probability to the more important units. We show that forcing the network to solve the task at hand in the absence of its important units yields a strong regularization effect. Further analysis indicates that Y-Drop yields solutions where more neurons are important, i.e have high conductance, and yields robust networks. In our experiments we show that the regularization effect of Y-Drop scales better than vanilla dropout w.r.t. the architecture size and consistently yields superior performance over multiple datasets and architecture combinations, with little tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。