稀疏神经网络在难样本学习上表现更优，尤其数据有限时。
Are Sparse Neural Networks Better Hard Sample Learners?
- 通过调整层间稀疏度比例，提升稀疏模型对难样本的适应能力。
- 在数据稀缺条件下，部分稀疏模型准确率超过密集模型。
- 适合关注高效训练与数据受限场景的研究者参考。
尽管深度学习取得了显著进展，但从难样本中学习仍是巨大挑战，因这些样本通常噪声多且结构复杂。难样本对深度神经网络的最优性能至关重要。现有针对稀疏神经网络（SNNs）的研究多聚焦于标准训练数据，缺乏对其在复杂、高难度数据上有效性的理解。本文在多种场景下的广泛实验表明，多数在难样本上训练的SNNs在特定稀疏度下可达到甚至超越密集模型的准确率，尤其在数据有限时。我们发现，层间密度比率在SNN性能中起关键作用，特别是从零开始训练而无预训练初始化的方法。这些发现深化了对SNN行为的理解，为数据驱动的人工智能高效学习方法提供了新思路。代码已公开于：https://github.com/QiaoXiao7282/hard_sample_learners。
原文摘要 · Abstract (English)
While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: \url{https://github.com/QiaoXiao7282/hard_sample_learners}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。