提出行为恢复秩量化神经网络功能冗余，发现剪枝后仍存大量自由度。
Functional Degeneracy in Neural Networks: Measurement and Pruning
- 用行为海森矩阵主方向数衡量模型可压缩性
- 结构剪枝和幅度剪枝在任务饱和后仍保留更多自由度
- 适合关注模型压缩与参数冗余的研究者
现代机器学习的核心问题之一是如何在不改变模型行为的前提下进行压缩，以降低部署时的内存、计算和能耗。为此，本文通过行为恢复秩（behavioral recovery rank）量化功能退化性，即恢复训练模型性能所需的前导行为-海森特征方向数量。以行为恢复秩为几何基准，研究发现结构剪枝和幅度剪枝即使在任务性能饱和后，仍保留更多自由度。这一差距表明，功能冗余分布于参数方向之间，而非由单个权重或神经元暴露。
原文摘要 · Abstract (English)
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。