用连接监控网络提升稀疏模型在分布偏移下的泛化能力
Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts
- 引入辅助网络GC-Net监测原网络层间连接
- 结合重要性与连接度剪枝，提升模型鲁棒性
- 适合需要高效且稳定推理的部署场景
稀疏深度神经网络（DNN）在机器人和计算机视觉等实际应用中表现优异，因其降低了计算开销。然而，现有方法多基于任务相关性剪除冗余神经元或滤波器，忽视了对分布偏移的适应能力。本文提出一种配套网络Ghost Connect-Net（GC-Net），通过其权重表征原网络相邻层间的连接强度，具备分布泛化优势。剪枝后，将剪枝位置映射回原网络，实现基于重要性和连接度的混合剪枝。在CIFAR-10、Fashion MNIST和Tiny ImageNet等基准测试中，该方法表现出色，尤其在深层网络中使用GC-Net指导剪枝、浅层直接剪枝的效果更优。论文还提供了理论基础，解释了该方法如何提升分布偏移下的泛化性能。
原文摘要 · Abstract (English)
Sparse deep neural networks (DNNs) excel in real-world applications like robotics and computer vision, by reducing computational demands that hinder usability. However, recent studies aim to boost DNN efficiency by trimming redundant neurons or filters based on task relevance, but neglect their adaptability to distribution shifts. We aim to enhance these existing techniques by introducing a companion network, Ghost Connect-Net (GC-Net), to monitor the connections in the original network with distribution generalization advantage. GC-Net's weights represent connectivity measurements between consecutive layers of the original network. After pruning GC-Net, the pruned locations are mapped back to the original network as pruned connections, allowing for the combination of magnitude and connectivity-based pruning methods. Experimental results using common DNN benchmarks, such as CIFAR-10, Fashion MNIST, and Tiny ImageNet show promising results for hybridizing the method, and using GC-Net guidance for later layers of a network and direct pruning on earlier layers. We provide theoretical foundations for GC-Net's approach to improving generalization under distribution shifts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。