提出统一框架,让模型压缩更平稳、准确率下降更少。
Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression
- 用渐进式融合原模型与压缩模型,避免突变
- 多数场景准确率提升1%以上,最高超15%
- 兼容各类压缩方法,适合追求稳定压缩的研究者
深度神经网络规模日益增大,亟需剪枝、量化和低秩分解等压缩技术以降低内存、计算和能耗。尽管这些方法有效,但常导致精度显著下降,通常通过迭代渐进压缩缓解。然而不同压缩方式需独立的迭代策略,且部分方法引发不稳定的微调过程。本文提出统一的渐逝贡献(VCON)框架,实现模型从原始形式向压缩形式的平滑、迭代过渡。在微调过程中,原模型与压缩模型并行运行,原模型贡献逐步衰减,压缩模型贡献逐渐增强。这种仿射组合使网络缓慢适应,提升稳定性并缓解精度损失。我们在计算机视觉和自然语言处理基准上评估了VCON,结合多种压缩策略。在多数设置下,其性能优于一次性压缩和传统迭代基线,典型提升超过1%,部分配置达15%以上。VCON可兼容现有压缩技术,并在多样化任务中持续提升表现。
原文摘要 · Abstract (English)
The increasing scale of Deep Neural Networks (DNNs) introduces the need for compression techniques such as pruning, quantization, and low-rank decomposition. While these methods are very effective at reducing memory, computation, and energy consumption, they may introduce severe accuracy degradation, which is often mitigated by using iterative, gradual compression. However, different compression techniques require distinct iterative approaches, and some result in unstable, discontinuous model fine-tuning. We introduce Vanishing Contributions (VCON), a unified framework for the smooth, iterative transition of DNNs into a compressed form. Rather than replacing the original network directly with its compressed version, VCON executes both in parallel during fine-tuning. The contribution of the original (uncompressed) model is progressively reduced, while that of the compressed model is gradually increased. This affine combination allows the network to slowly adapt, improving stability and mitigating accuracy degradation. We evaluate VCON on computer vision and natural language processing benchmarks, using multiple compression strategies. In most settings, our framework improves accuracy over post-shot and iterative baselines. Typical gains exceed 1%, while some configuration exhibits improvements above 15%. VCON is thus compatible with existing compression techniques and consistently improves performance across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。