arXiv:2511.09901cs.LG2025-11AAAI

同时剪枝权重和精选数据,让模型更小更快还更准。

Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training

  • 交替剪裁权重和筛选数据,形成协同优化机制
  • 联合优化可提升准确率最高达17.83%,减少10%~90%计算量
  • 提出状态保持机制,避免关键权重与样本被误删

现代深度神经网络依赖大量模型参数和训练样本,带来巨大计算开销。权重剪枝和核心集选择是两种提升计算效率的新范式。本文首次通过透明分析揭示冗余权重与训练样本之间的相互作用:噪声样本导致权重过度拟合,使剪枝难以识别无关权重;而无关权重又会过拟合噪声数据,降低核心集选择效果。为此,我们提出同步权重与样本裁剪机制(SWaST),交替执行权重剪枝与核心集选择,以建立协同效应。实验中发现,当同时大规模删除权重和样本时,会出现‘关键双损失’现象——重要权重及其支持样本被误删,导致模型失稳且无法恢复。这在深度学习中尤为突出,因缺乏剪枝与核心集选择的理论保障,致使两者常独立发展。我们通过在SWaST中引入状态保持机制,实现稳定联合优化。大量实验表明,在不同剪枝率与核心集规模下,二者存在强协同效应,准确率最高提升17.83%,同时实现10%至90%的浮点运算量(FLOPs)降低。

原文摘要 · Abstract (English)

Modern deep neural networks rely heavily on massive model weights and training samples, incurring substantial computational costs. Weight pruning and coreset selection are two emerging paradigms proposed to improve computational efficiency. In this paper, we first explore the interplay between redundant weights and training samples through a transparent analysis: redundant samples, particularly noisy ones, cause model weights to become unnecessarily overtuned to fit them, complicating the identification of irrelevant weights during pruning; conversely, irrelevant weights tend to overfit noisy data, undermining coreset selection effectiveness. To further investigate and harness this interplay in deep learning, we develop a Simultaneous Weight and Sample Tailoring mechanism (SWaST) that alternately performs weight pruning and coreset selection to establish a synergistic effect in training. During this investigation, we observe that when simultaneously removing a large number of weights and samples, a phenomenon we term critical double-loss can occur, where important weights and their supportive samples are mistakenly eliminated at the same time, leading to model instability and nearly irreversible degradation that cannot be recovered in subsequent training. Unlike classic machine learning models, this issue can arise in deep learning due to the lack of theoretical guarantees on the correctness of weight pruning and coreset selection, which explains why these paradigms are often developed independently. We mitigate this by integrating a state preservation mechanism into SWaST, enabling stable joint optimization. Extensive experiments reveal a strong synergy between pruning and coreset selection across varying prune rates and coreset sizes, delivering accuracy boosts of up to 17.83% alongside 10% to 90% FLOPs reductions.

模型压缩权重剪枝核心集选择协同优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。