通过调整稀疏掩码对齐优化盆地，提升随机初始化下的稀疏训练泛化能力。
Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
- 用权重对称性对齐不同初始化的优化盆地，使稀疏掩码可迁移。
- 在多个数据集和模型上，使用对齐掩码后泛化性能显著提升。
- 适合关注稀疏训练迁移性与高效模型设计的研究者。
彩票理论假设存在一个稀疏的掩码和权重组合，能在参数大幅减少的情况下实现与稠密模型相当的泛化性能。然而，找到此类解计算成本高，且彩票掩码无法推广到新的随机初始化。近期研究表明，从随机初始化训练的神经网络解位于同一损失盆地（模置换），并提出方法对齐同一损失盆地内的模型。本文假设，盆地错位是导致彩票掩码无法跨初始化通用的原因，提出在不同随机初始化下进行稀疏训练时，通过置换彩票掩码以对齐新的优化盆地。实验表明，在CIFAR-10、CIFAR-100和ImageNet等多个数据集，以及VGG11、ResNet20和ResNet50等模型上，使用经过置换的掩码进行稀疏训练，相比未置换掩码，泛化性能有显著提升。
原文摘要 · Abstract (English)
The Lottery Ticket Hypothesis (LTH) suggests there exists a sparse LTH mask and weights that achieve the same generalization performance as the dense model while using significantly fewer parameters. However, finding a LTH solution is computationally expensive, and a LTH sparsity mask does not generalize to other random weight initializations. Recent work has suggested that neural networks trained from random initialization find solutions within the same basin modulo permutation, and proposes a method to align trained models within the same loss basin. We hypothesize that misalignment of basins is the reason why LTH masks do not generalize to new random initializations and propose permuting the LTH mask to align with the new optimization basin when performing sparse training from a different random init. We empirically show a significant increase in generalization when sparse training from random initialization with the permuted mask as compared to using the non-permuted LTH mask, on multiple datasets (CIFAR-10, CIFAR-100 and ImageNet) and models (VGG11, ResNet20 and ResNet50).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。