arXiv:2509.25665cs.LG2025-09中稿 · ICANN 2026

提出渐进式扩容方法,自动发现高效稀疏网络。

Gradual Capacity Growth for Sparse Network Discovery

  • 从稀疏网络开始,按信号路径逐步添加连接。
  • 在中等密度下逼近稠密网络性能,节省训练成本。
  • 适合资源有限时探索模型精度与规模的平衡。

稀疏神经网络方法通常预设目标稀疏度,但网络容量与性能的关系往往未知且依赖任务。现有方法(如迭代剪枝、动态稀疏训练、初始化剪枝)或依赖稠密预训练,或带来高昂重训成本,或需预设稀疏预算。本文提出渐进式容量增长(GCG),一种从稀疏到稠密的构建式训练框架,通过PathGrow规则分阶段增加连接,优先选择高信号输入输出路径,同时保持结构多样性。增长过程穿插有限训练,并采用轻量级性能-密度外推规则,估计进一步扩容收益递减的最小密度。在CIFAR、TinyImageNet和ImageNet上的实验表明,GCG能高效发现稀疏网络,在中等密度下实现接近稠密网络的性能,无需稠密预训练或每层稀疏度的反复重训。虽无法达到剪枝或动态重分配的极低稀疏度,但相比迭代幅度剪枝显著降低总训练成本,为优化预算受限场景下的精度-密度权衡提供了实用机制。

原文摘要 · Abstract (English)

Sparse neural network methods typically assume that the target sparsity (or density) is fixed in advance, even though the relationship between network capacity and performance is generally unknown and task-dependent. Existing approaches (including iterative pruning, dynamic sparse training, and pruning at initialization) either rely on dense pretraining, incur substantial retraining cost, or require a preset sparsity budget. We propose Gradual Capacity Growth (GCG), a constructive sparse-to-dense training framework that allocates network capacity progressively during training. Starting from a sparse seed network, GCG grows new connections in stages using PathGrow, a probabilistic path-based growth rule that biases additions toward high-signal input-output pathways while preserving structural diversity. Growth is interleaved with limited training, and a lightweight performance-density extrapolation rule is used to estimate the smallest density beyond which further capacity increases yield diminishing accuracy gains. Experiments on CIFAR, TinyImageNet, and ImageNet show that GCG efficiently identifies sparse networks that achieve near-dense performance at moderate densities, without requiring dense pretraining or exhaustive retraining at each sparsity level. While growth-only methods do not reach the extreme sparsity achievable by pruning or dynamic reallocation techniques, GCG substantially reduces total training cost compared to iterative magnitude pruning and provides a practical mechanism for exploring the accuracy-density tradeoff under limited optimization budgets.

稀疏网络模型压缩训练效率容量增长

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。