arXiv:2409.04757cs.LG2024-09

通过自适应聚类提升神经网络梯度稳定性,实现更高效训练。

Unsupervised Adaptive Normalization

  • 用高斯混合模型对激活值聚类,动态生成归一化参数。
  • 相比传统方法,训练速度更快且分类/域适应性能更好。
  • 无需标签,适合对梯度不稳敏感的任务场景。

深度神经网络在解决复杂问题上已成标配,但反向传播中激活分布的漂移常导致梯度不稳定。批归一化(BN)虽能缓解此问题并支持更高学习率,却依赖小批量大小且假设样本分布均匀。为此,已有层归一化、组归一化等替代方案,但仍难以适应神经元激活的动态分布。本文提出无监督自适应归一化(UAN),将聚类与归一化统一于单一学习过程:利用高斯混合模型对激活值进行聚类,为每个簇确定归一化参数,并随网络权重同步更新,以适配目标任务。该联合机制基于激活值归一化,形成任务定制的自适应表示,增强梯度稳定性,加快学习进程,提升模型性能。UAN在分类与域适应任务中均优于经典方法。

原文摘要 · Abstract (English)

Deep neural networks have become a staple in solving intricate problems, proving their mettle in a wide array of applications. However, their training process is often hampered by shifting activation distributions during backpropagation, resulting in unstable gradients. Batch Normalization (BN) addresses this issue by normalizing activations, which allows for the use of higher learning rates. Despite its benefits, BN is not without drawbacks, including its dependence on mini-batch size and the presumption of a uniform distribution of samples. To overcome this, several alternatives have been proposed, such as Layer Normalization, Group Normalization, and Mixture Normalization. These methods may still struggle to adapt to the dynamic distributions of neuron activations during the learning process. To bridge this gap, we introduce Unsupervised Adaptive Normalization (UAN), an innovative algorithm that seamlessly integrates clustering for normalization with deep neural network learning in a singular process. UAN executes clustering using the Gaussian mixture model, determining parameters for each identified cluster, by normalizing neuron activations. These parameters are concurrently updated as weights in the deep neural network, aligning with the specific requirements of the target task during backpropagation. This unified approach of clustering and normalization, underpinned by neuron activation normalization, fosters an adaptive data representation that is specifically tailored to the target task. This adaptive feature of UAN enhances gradient stability, resulting in faster learning and augmented neural network performance. UAN outperforms the classical methods by adapting to the target task and is effective in classification, and domain adaptation.

归一化聚类自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。