用可微分门控机制高效发现稀疏高精度初始神经网络子结构。
Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates
- 通过连续松弛伯努利门控实现端到端可微优化，无需迭代剪枝。
- 在多种模型上实现90%稀疏度，准确率损失极小，是现有方法的两倍。
- 适合需要轻量化部署的场景，尤其对资源受限设备友好。
过参数化神经网络在资源受限环境下部署时面临内存与计算成本过高的问题。强彩票券（SLT）假说指出，随机初始化的网络中存在稀疏子网络，在不进行权重训练的情况下即可达到竞争性精度。现有SLT方法（如edge-popup）依赖非可微的基于分数的选择策略，限制了优化效率与可扩展性。本文提出使用连续松弛伯努利门控，通过完全可微、端到端的方式发现SLT——仅训练门控参数，冻结所有网络权重在初始化值。连续松弛使直接优化ℓ₀正则化目标成为可能，避免了非可微梯度估计器或迭代剪枝循环。据我们所知，这是首个无需直通估计器近似的一致可微SLT发现方法。在全连接网络、CNN（ResNet、Wide-ResNet）和视觉变换器（ViT、Swin-T）上的实验表明，可实现高达90%的稀疏度，且准确率损失极小，性能接近边弹法（edge-popup）的两倍，建立了一种可扩展的预训练网络稀疏化框架。
原文摘要 · Abstract (English)
Over-parameterized neural networks incur prohibitive memory and computational costs for resource-constrained deployment. The Strong Lottery Ticket (SLT) hypothesis suggests that randomly initialized networks contain sparse subnetworks achieving competitive accuracy without weight training. Existing SLT methods, notably edge-popup, rely on non-differentiable score-based selection, limiting optimization efficiency and scalability. We propose using continuously relaxed Bernoulli gates to discover SLTs through fully differentiable, end-to-end optimization - training only gating parameters while keeping all network weights frozen at their initialized values. Continuous relaxation enables direct gradient-based optimization of an $\ell_0$-regularization objective, eliminating the need for non-differentiable gradient estimators or iterative pruning cycles. To our knowledge, this is the first fully differentiable approach for SLT discovery that avoids straight-through estimator approximations. Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss - nearly double the sparsity achieved by edge-popup at comparable accuracy - establishing a scalable framework for pre-training network sparsification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。