用李雅普诺夫谱加速剪枝搜索,找到比原始模型还准的轻量版网络。
Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum
- 基于李雅普诺夫谱设计距离度量,训练前就能预测剪枝后性能。
- 在固定训练预算下,剪枝模型性能超越原始模型和传统方法选型。
- 适用于追求高效轻量化RNN的开发者,尤其适合资源受限场景。
针对过参数化的循环神经网络(RNN),已有多种剪枝方法可降低功耗与存储占用。这些进展催生了「超剪枝」新范式,旨在为特定网络架构与应用选择最优剪枝策略。不同于传统超参搜索中目标精度不确定的问题,在剪枝场景中,密集模型的准确率是剪枝模型的性能基准。目标即寻找能达到甚至超越该基准的剪枝变体。然而,对所有剪枝配置进行穷举搜索计算成本高且缺乏早期性能保证。为此,本文提出一种基于李雅普诺夫谱(Lyapunov Spectrum, LS)的距离度量,可在训练初期实现剪枝与密集网络的快速比较,从而准确预测训练后的性能表现。结合标准超参优化算法,我们构建了高效的超剪枝框架——基于李雅普诺夫谱的超剪枝(LS-based Hyperpruning, LSH)。实验在堆叠LSTM与RHN架构上使用Penn Treebank数据集,以及AWD-LSTM-MoS在WikiText-2上的结果表明,在固定训练预算和目标剪枝率条件下,LSH能持续发现更优的剪枝模型。尤为显著的是,这些剪枝变体不仅优于基于损失的基线方法选出的模型,甚至超越其原始密集模型的性能。
原文摘要 · Abstract (English)
A variety of pruning methods have been introduced for over-parameterized Recurrent Neural Networks to improve efficiency in terms of power consumption and storage utilization. These advances motivate a new paradigm, termed `hyperpruning', which seeks to identify the most suitable pruning strategy for a given network architecture and application. Unlike conventional hyperparameter search, where the optimal configuration's accuracy remains uncertain, in the context of network pruning, the accuracy of the dense model sets the target for the accuracy of the pruned one. The goal, therefore, is to discover pruned variants that match or even surpass this established accuracy. However, exhaustive search over pruning configurations is computationally expensive and lacks early performance guarantees. To address this challenge, we propose a novel Lyapunov Spectrum (LS)-based distance metric that enables early comparison between pruned and dense networks, allowing accurate prediction of post-training performance. By integrating this LS-based distance with standard hyperparameter optimization algorithms, we introduce an efficient hyperpruning framework, termed LS-based Hyperpruning (LSH). LSH reduces search time by an order of magnitude compared to conventional approaches relying on full training. Experiments on stacked LSTM and RHN architectures using the Penn Treebank dataset, and on AWD-LSTM-MoS using WikiText-2, demonstrate that under fixed training budgets and target pruning ratios, LSH consistently identifies superior pruned models. Remarkably, these pruned variants not only outperform those selected by loss-based baseline but also exceed the performance of their dense counterpart.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。