用激活值指导自动编码器剪枝,提升效率与性能。
Guiding Evolutionary AutoEncoder Training with Activation-Based Pruning Operators
- 基于层激活设计新型变异算子,引导权重剪枝。
- 在单个模型中剪枝后性能接近原始模型,效率更高。
- 适合需要高效压缩的自动编码器应用者。
本研究提出一种基于进化计算的神经网络剪枝新方法,聚焦于同时剪枝自动编码器的编码器与解码器。引入两种利用层激活信息指导权重剪枝的新变异算子。实验表明,其中一种激活引导算子优于随机剪枝,生成的自动编码器更高效且性能接近标准训练模型。先前研究已证明,采用空间协同进化算法协同演化编码器与解码器种群,比单个自动编码器更有效且可扩展。本文评估相同激活引导算子在该协同进化框架中的表现,发现随机剪枝反而优于引导剪枝。这表明激活引导在低维剪枝环境中更有效,因受限样本空间可能导致随机化偏离真实均匀性;而种群驱动策略通过扩大总剪枝维度,实现统计上的均匀随机性,更好地保持系统动态。研究还测试了不同剪枝调度策略,给出了针对标准与协同进化种群的最佳算子与调度组合。
原文摘要 · Abstract (English)
This study explores a novel approach to neural network pruning using evolutionary computation, focusing on simultaneously pruning the encoder and decoder of an autoencoder. We introduce two new mutation operators that use layer activations to guide weight pruning. Our findings reveal that one of these activation-informed operators outperforms random pruning, resulting in more efficient autoencoders with comparable performance to canonically trained models. Prior work has established that autoencoder training is effective and scalable with a spatial coevolutionary algorithm that cooperatively coevolves a population of encoders with a population of decoders, rather than one autoencoder. We evaluate how the same activity-guided mutation operators transfer to this context. We find that random pruning is better than guided pruning, in the coevolutionary setting. This suggests activation-based guidance proves more effective in low-dimensional pruning environments, where constrained sample spaces can lead to deviations from true uniformity in randomization. Conversely, population-driven strategies enhance robustness by expanding the total pruning dimensionality, achieving statistically uniform randomness that better preserves system dynamics. We experiment with pruning according to different schedules and present best combinations of operator and schedule for the canonical and coevolving populations cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。