用偏序集建模RNN结构,实现更精准的剪枝。
Lattice-Based Pruning in Recurrent Neural Networks via Poset Modeling
- 将RNN视为偏序集,构建依赖格,识别关键神经元。
- 中等剪枝保持98%以上准确率,激进剪枝提升稀疏度。
- 适合追求高效深层网络与结构化剪枝的研究者。
循环神经网络(RNN)在序列建模中至关重要,但其高计算复杂性制约了可扩展性和实时部署。传统基于权重大小的剪枝方法常忽略网络内在结构特性。本文提出一种新框架,将RNN建模为偏序集(posets),构建对应的依赖格。通过识别交不可约神经元,所提格剪枝算法选择性保留关键连接,同时移除冗余连接。该方法采用二值和连续邻接矩阵来捕捉网络连通性的不同方面。在MNIST数据集上的评估显示,剪枝存在稀疏度与分类精度间的明确权衡:中等剪枝下准确率保持在98%以上,激进剪枝则实现更高稀疏度,仅伴随小幅性能下降。相较于传统基于幅度的剪枝,本方法利用了RNN的结构性组织,更有效地保留功能连通性,并提升了具有自上而下反馈的多层网络的效率。所提出的格剪枝框架为降低RNN复杂性提供了严谨且可扩展的方法,有助于实现更高效的层次化模型,在机器学习与计算神经科学中具有前景。
原文摘要 · Abstract (English)
Recurrent neural networks (RNNs) are central to sequence modeling tasks, yet their high computational complexity poses challenges for scalability and real-time deployment. Traditional pruning techniques, predominantly based on weight magnitudes, often overlook the intrinsic structural properties of these networks. We introduce a novel framework that models RNNs as partially ordered sets (posets) and constructs corresponding dependency lattices. By identifying meet irreducible neurons, our lattice-based pruning algorithm selectively retains critical connections while eliminating redundant ones. The method is implemented using both binary and continuous-valued adjacency matrices to capture different aspects of network connectivity. Evaluated on the MNIST dataset, our approach exhibits a clear trade-off between sparsity and classification accuracy. Moderate pruning maintains accuracy above 98%, while aggressive pruning achieves higher sparsity with only a modest performance decline. Unlike conventional magnitude-based pruning, our method leverages the structural organization of RNNs, resulting in more effective preservation of functional connectivity and improved efficiency in multilayer networks with top-down feedback. The proposed lattice-based pruning framework offers a rigorous and scalable approach for reducing RNN complexity while sustaining robust performance, paving the way for more efficient hierarchical models in both machine learning and computational neuroscience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。