arXiv:2511.16061cs.LG2025-11

提出旋转不变激活函数,实现无需额外参数的结构化剪枝

Change-of-Basis Pruning via Rotational Invariance

  • 设计双子空间径向激活函数,使网络对正交变换保持不变
  • 在固定比例剪枝下将可靠剪枝范围从30%扩展至70%
  • 适合追求高效模型压缩且可接受轻微精度损失的研究者

结构化剪枝通过移除整个神经元或通道来压缩模型,但其效果依赖于重要性在表示空间中的分布。换基(CoB)剪枝通过施加正交线性变换,将重要性集中于特定维度。然而,多数主流深度学习架构不天然具备此类变换的不变性。为此,本文提出双子空间径向激活函数(TSRAs),其在两个激活子空间内独立应用正交变换时保持不变。该不变性使CoB变换可合并到邻近权重中,无需增加参数。本工作作为概念验证,表明旋转不变设计可能为换基剪枝提供理论基础。虽未分析多种TSRA候选或优化初始化,且为实现旋转不变性进行了必要修改,导致相比ReLU基线精度下降4.52%,但在CIFAR-10上,采用激活幅度重要性的VGG-16模型表现良好:在固定比例剪枝下,所有剪枝率下性能均优于基线,可靠剪枝范围由约30%扩展至70%;在阈值剪枝策略下,可剪除90%-96%参数,微调后仅损失1%-6%精度。结果表明,旋转不变架构为CoB剪枝提供了有前景的方向。

原文摘要 · Abstract (English)

Structured pruning removes entire neurons or channels, but its effectiveness depends on how importance is distributed across the representation space. Change-of-basis (CoB) pruning addresses this challenge by applying orthogonal linear transformations that concentrate importance within certain dimensions. However, many standard deep learning architectures are not inherently invariant to such transformations. To enable compatibility, we introduce two-subspace radial activations (TSRAs): an activation family that is invariant to orthogonal linear transformations applied independently within its two activation subspaces. This invariance allows CoB transformations to be merged into surrounding weights without incurring extra parameters. We position this work as a proof-of-concept that a rotationally invariant design may offer a principled approach towards change-of-basis pruning. We do not provide an analysis of multiple TSRA candidates nor do we explore weight initialization for any TSRAs. These limitations, combined with other necessary modifications we make to permit rotational invariance, result in a slight accuracy drop of $4.52\%$ compared to a ReLU-based control. However, using activation-magnitude importance, VGG-16 implementing our CoB+TSRA framework shows encouraging results on CIFAR-10. Under fixed-ratio structured pruning, CoB improves accuracy over a TSRA baseline at all pruning ratios and extends reliable pruning frontier from roughly $30\%$ to $70\%$ of parameters without post-prune fine tuning. Under threshold-based pruning strategies, CoB prunes $90-96\%$ of parameters while maintaining $1-6\%$ accuracy drop after fine-tuning. Together, these results indicate that rotationally invariant architectures may offer a promising path towards CoB pruning.

结构化剪枝旋转不变性模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。