揭示剪枝如何改变深度网络的交互模式，解释为何某些参数剪枝会严重降性能。
How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration
- 通过分析网络内部交互模式变化，发现剪枝影响分三个阶段。
- 低阶交互模式具有强泛化能力，其破坏导致性能显著下降。
- 适合关注模型鲁棒性与结构优化的研究者阅读。
本研究聚焦于理解深度神经网络（DNN）在不同参数剪枝时性能差异的内在机制。为解释为何剪除某些参数会导致显著性能下降而其他参数则不会，我们考察剪枝操作如何影响DNN编码的交互模式。研究发现，随着剪枝率逐步提高，DNN编码的交互模式呈现出明显的三阶段动态：在剪枝开始移除低阶交互前，模型性能基本不受影响；低阶交互模式表现出强泛化能力。此外，我们发现某些模块对剪枝高度敏感，原因在于剪枝是否破坏了可泛化的低阶交互模式。
原文摘要 · Abstract (English)
This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, \emph{i.e.}, model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。