通过信息流差异性分析,同时剪裁冗余滤波器和层,实现高效模型压缩。
IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization
- 基于张量信息流差异性,分两阶段剪裁滤波器与层。
- 在多种架构上实现显著参数压缩,精度保持竞争力。
- 适用于卷积、Transformer等结构,适合资源受限部署。
本文提出一种新型神经网络压缩方法,通过统一框架在滤波器与网络结构层面消除冗余,其基础是信息流分析。基于张量流差异性(tensor flow divergence)量化信息在各层间的转换程度,设计了两阶段优化流程:第一阶段采用迭代式差异感知剪枝,识别并移除冗余滤波器,同时保留关键信息路径;第二阶段将该原则拓展至更高层级,通过分析各层对信息传播的贡献,选择性删除对性能影响最小的完整层。该方法可自然适配多种架构,包括卷积网络、Transformer及混合设计,提供跨层类型的一致结构重要性度量标准。在多个现代架构与数据集上的实验表明,该方法在实现显著模型压缩的同时保持优异精度,参数减少效果达到甚至超越当前最优水平,在从卷积模型到Transformer的广泛架构中表现优异。结果证明,信息流差异性可有效指导滤波器与层级别的优化,为资源受限环境下的实际部署提供实用价值。
原文摘要 · Abstract (English)
This paper presents a novel approach to neural network compression that addresses redundancy at both the filter and architectural levels through a unified framework grounded in information flow analysis. Building on the concept of tensor flow divergence, which quantifies how information is transformed across network layers, we develop a two-stage optimization process. The first stage employs iterative divergence-aware pruning to identify and remove redundant filters while preserving critical information pathways. The second stage extends this principle to higher-level architecture optimization by analyzing layer-wise contributions to information propagation and selectively eliminating entire layers that demonstrate minimal impact on network performance. The proposed method naturally adapts to diverse architectures, including convolutional networks, transformers, and hybrid designs, providing a consistent metric for comparing the structural importance across different layer types. Experimental validation across multiple modern architectures and datasets reveals that this combined approach achieves substantial model compression while maintaining competitive accuracy. The presented approach achieves parameter reduction results that are globally comparable to those of state-of-the-art solutions and outperforms them across a wide range of modern neural network architectures, from convolutional models to transformers. The results demonstrate how flow divergence serves as an effective guiding principle for both filter-level and layer-level optimization, offering practical benefits for deployment in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。