arXiv:2512.12663cs.LGcs.AI2025-12

让每个神经元在不同样本中随机失活,提升模型泛化能力

PerNodeDrop: A Method Balancing Specialized Subnets and Regularization in Deep Neural Networks

  • 针对每一样本、每个神经元独立施加随机失活,打破噪声均匀性
  • 在图像、文本、音频任务上显著缩小训练与验证性能差距
  • 适合需要强泛化性的深度模型,尤其对过拟合敏感场景

深度神经网络具有强大的表征能力,但易过拟合,因神经元常协同适应,既捕捉复杂特征交互,也强化虚假非泛化模式,导致训练表现高而测试可靠性低。现有基于噪声的正则化方法如Dropout和DropConnect通过训练时注入随机扰动缓解此问题,但其噪声通常层内或批次内均匀分布,会同时抑制有害与有益的协同适应。本文提出PerNodeDrop,一种轻量级随机正则化方法,在样本层面、节点层面施加扰动,使每个神经元经历输入相关的变异性,从而保留有用协同适应的同时实现正则化。实验表明,该方法有效缩小训练与验证性能差距,提升未见数据上的可靠性。尽管表面类似DropConnect,但其在样本层面而非批次层面丢弃权重。期望损失分析证明其能削弱过度协同适应,同时保留预测性交互。在视觉、文本、音频基准测试中,其泛化性能优于标准噪声正则化方法。

原文摘要 · Abstract (English)

Deep neural networks possess strong representational capacity yet remain vulnerable to overfitting, primarily because neurons tend to co-adapt in ways that, while capturing complex and fine-grained feature interactions, also reinforce spurious and non-generalizable patterns that inflate training performance but reduce reliability on unseen data. Noise-based regularizers such as Dropout and DropConnect address this issue by injecting stochastic perturbations during training, but the noise they apply is typically uniform across a layer or across a batch of samples, which can suppress both harmful and beneficial co-adaptation. This work introduces PerNodeDrop, a lightweight stochastic regularization method. It applies per-sample, per-node perturbations to break the uniformity of the noise injected by existing techniques, thereby allowing each node to experience input-specific variability. Hence, PerNodeDrop preserves useful co-adaptation while applying regularization. This narrows the gap between training and validation performance and improves reliability on unseen data, as evident from the experiments. Although superficially similar to DropConnect, PerNodeDrop operates at the sample level. It drops weights at the sample level, not the batch level. An expected-loss analysis formalizes how its perturbations attenuate excessive co-adaptation while retaining predictive interactions. Empirical evaluations on vision, text, and audio benchmarks indicate improved generalization relative to the standard noise-based regularizer.

正则化过拟合深度学习泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。