80%连接剪枝后,受限玻尔兹曼机仍能生成高质量数据,但后期剪枝无法恢复性能。
Diluting Restricted Boltzmann Machines
- 在训练前剪枝80%连接,仍能保留高质量生成能力
- 剪枝超过临界点后生成质量突然下降,存在性能断崖
- 后期剪枝的模型表现不如从头训练的同稀疏度模型
人工智能发展依赖于日益庞大的神经网络,带来计算与环境成本压力。本文研究在极端剪枝条件下受限玻尔兹曼机(RBMs)的表现。受彩票理论启发,我们发现即使训练前剪枝80%连接,RBMs仍可实现高质量生成,表明其包含可行子网络。然而实验揭示关键局限:一旦额外剪枝,训练后的网络无法通过再训练恢复性能。当剪枝破坏最小核心连接时,生成质量出现急剧下降。此外,重新训练的剪枝网络仍受限于初始参数,表现劣于同稀疏度下从头训练的模型。结果表明,为使稀疏网络有效,剪枝应尽早进行而非事后补救。研究为高效神经架构设计提供实践指导,并强调初始条件对网络能力的持久影响。
原文摘要 · Abstract (English)
Recent advances in artificial intelligence have relied heavily on increasingly large neural networks, raising concerns about their computational and environmental costs. This paper investigates whether simpler, sparser networks can maintain strong performance by studying Restricted Boltzmann Machines (RBMs) under extreme pruning conditions. Inspired by the Lottery Ticket Hypothesis, we demonstrate that RBMs can achieve high-quality generative performance even when up to 80% of the connections are pruned before training, confirming that they contain viable sub-networks. However, our experiments reveal crucial limitations: trained networks cannot fully recover lost performance through retraining once additional pruning is applied. We identify a sharp transition above which the generative quality degrades abruptly when pruning disrupts a minimal core of essential connections. Moreover, re-trained networks remain constrained by the parameters originally learned performing worse than networks trained from scratch at equivalent sparsity levels. These results suggest that for sparse networks to work effectively, pruning should be implemented early in training rather than attempted afterwards. Our findings provide practical insights for the development of efficient neural architectures and highlight the persistent influence of initial conditions on network capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。