用共识准则剪枝层,模型更小更快更抗攻击。
Layer Pruning with Consensus: A Triple-Win Solution
- 融合多种相似性度量,生成统一的低重要性层判断标准。
- 最高减少78.80%计算量,性能接近顶尖方法,碳排放降68.75%。
- 提升模型鲁棒性,适合追求高效低碳部署的场景。
层剪枝为降低计算成本、延迟和内存占用提供了有前景的替代方案。尽管已有方法尝试识别可移除的不重要层,但多依赖单一标准,难以全面捕捉层的复杂特性。本文提出一种新方法,将多种相似性度量融合为统一的低重要性层判别标准——共识准则。该方法实现三重收益:准确率下降小、性能提升显著、对对抗攻击更具鲁棒性。在多个基准上,最多可减少78.80% FLOPs,性能与当前最优方法相当;同时能源消耗和碳排放分别降低高达66.99%和68.75%。此外,有效避免捷径学习,在各类对抗攻击下鲁棒性提升最高达4个百分点。整体表明,共识准则能构建高效、稳健且环保的剪枝模型。
原文摘要 · Abstract (English)
Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。