提出双评分方法,更可靠地找出无需训练就准确的神经网络子结构。
Double-Scoring: Reliable Extraction of Strong Lottery Tickets

- 用增强评分空间替代逐层稀疏度搜索,提升寻找强彩票票券的效率。
- 在固定稀疏度下,提取成功率显著高于现有方法,且对稀疏度参数不敏感。
- 适合研究模型压缩、彩票票券理论或高效初始化的科研人员参考。
彩票票券假设指出,大型随机神经网络中包含稀疏子网络,在经过类似训练后可达到密集模型性能。更强版本认为,充分过参数化的随机网络中存在子网络,无需权重训练即已具备高精度。已有理论证明此类强彩票票券存在,但可靠提取仍具挑战。本文重新审视冻结权重评分-训练方法edge-popup,识别出逐层稀疏度选择是核心瓶颈。提出双评分(double-scoring),通过扩增评分空间参数化,将逐层稀疏度搜索转化为在扩展评分张量上的优化。证明在扩增评分空间中固定密度掩码可保留原始坐标掩码的访问能力,并表明该方法可解释为在零扩增网络上运行的edge-popup。控制实验显示,双评分在固定密度下显著优于edge-popup与初始化剪枝基线,性能超越回溯稀疏训练拓扑,且对稀疏度超参数敏感度大幅降低。消融实验表明,性能提升不仅来自额外可训练评分参数,更源于扩增评分空间中的竞争机制所诱导的有效原始稀疏性。
原文摘要 · Abstract (English)
The lottery ticket hypothesis proposes that large random neural networks contain sparse subnetworks that can match the performance of dense models after comparable training. A stronger version asserts that sufficiently overparameterized random networks contain subnetworks that are already accurate before any weight training. Existing theory establishes that such strong lottery tickets exist, but reliable extraction remains difficult. We revisit edge-popup, a frozen-weight score-training method for extracting strong tickets, and identify layerwise sparsity selection as a central bottleneck. We introduce double-scoring, an augmented score-space parameterization that replaces a layerwise sparsity search with optimization over enlarged score tensors. We prove that fixed-density masking in an augmented score space preserves access to all original-coordinate masks, and we show that the resulting method can be interpreted as edge-popup on a zero-augmented network. In controlled experiments, double-scoring substantially improves strong-ticket extraction over fixed-density edge-popup and pruning-at-initialization baselines, improves on the performance of rewound sparse-training topologies, and exhibits markedly lower sensitivity to sparsity hyperparameters. Ablations show that the gain is not merely due to additional trainable score parameters, but is tied to the augmented score-space competition that induces the effective original sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。