提出新度量方法,解决深度网络结构剪枝中的功能路径丢失问题。
Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks
- 基于交替梯度流思想,用特征空间泰勒展开捕捉结构动态价值。
- 在ImageNet-1K上75%压缩下避免结构坍塌,优于传统方法。
- 适用于视觉变压器的动态推理,减少50%专家调用成本。
高效深度学习通常依赖静态启发式方法,如权重幅度或激活感知(如Wanda、RIA)。然而,在深层视觉网络的结构化剪枝中,这些方法存在幅度偏差,难以保留关键功能路径。为此,我们提出受交替梯度流(AGF)启发的解耦动力学范式,采用绝对特征空间泰勒展开精确捕捉网络的结构‘动能效用’。首先,我们在极端稀疏下发现拓扑相变,AGF成功维持基线性能并表现出拓扑隐式正则化,避免了从头训练模型的崩溃。其次,在无严格结构先验的架构中,我们揭示了视觉变压器(ViTs)存在稀疏瓶颈现象:收敛模型中动态信号因信号压缩而次优,影响实时路由。最后,基于这些实证约束,我们设计了一种混合路由框架,将AGF指导的离线结构搜索与零成本物理先验下的在线执行解耦。在大规模基准测试中验证:在ImageNet-1K上75%压缩压力下,AGF有效避免结构坍塌,而传统方法显著低于随机采样;系统部署于ImageNet-100动态推理时,该方法实现帕累托最优效率,使重型专家使用减少约50%(整体成本降至0.92×),且不牺牲全模型精度。
原文摘要 · Abstract (English)
Efficient deep learning traditionally relies on static heuristics like weight magnitude or activation awareness (e.g., Wanda, RIA). While successful in unstructured settings, we observe a critical limitation when applying these metrics to the structural pruning of deep vision networks. These contemporary metrics suffer from a magnitude bias, failing to preserve critical functional pathways. To overcome this, we propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF), utilizing an absolute feature-space Taylor expansion to accurately capture the network's structural "kinetic utility". First, we uncover a topological phase transition at extreme sparsity, where AGF successfully preserves baseline functionality and exhibits topological implicit regularization, avoiding the collapse seen in models trained from scratch. Second, transitioning to architectures without strict structural priors, we reveal a phenomenon of Sparsity Bottleneck in Vision Transformers (ViTs). Through a gradient-magnitude decoupling analysis, we discover that dynamic signals suffer from signal compression in converged models, rendering them suboptimal for real-time routing. Finally, driven by these empirical constraints, we design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors. We validate our paradigm on large-scale benchmarks: under a 75% compression stress test on ImageNet-1K, AGF effectively avoids the structural collapse where traditional metrics aggressively fall below random sampling. Furthermore, when systematically deployed for dynamic inference on ImageNet-100, our hybrid approach achieves Pareto-optimal efficiency. It reduces the usage of the heavy expert by approximately 50% (achieving an estimated overall cost of 0.92$\times$) without sacrificing the full-model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。