通过约束条件数与Lipschitz常数,提升高压缩率下模型的鲁棒性。
Lipschitz Constant Meets Condition Number: Learning Robust and Compact Deep Neural Networks
- 设计TSCNC联合约束,调控权重分布以降低条件数。
- 高压缩率下模型抗干扰能力显著提升,对抗攻击准确率提高。
- 理论揭示条件数是过度稀疏导致脆弱性的关键因素,适合模型压缩研究者。
近期研究发现,深度神经网络(DNN)的高密度压缩(如大幅剪枝权重矩阵)会导致精度下降和易受对抗攻击。将剪枝融入对抗训练框架可提升鲁棒性,但高度剪枝的权重矩阵往往病态,即条件数升高,加剧对输入噪声的敏感性。尽管高度剪枝可降低局部Lipschitz常数上界以容忍大扰动,但其病态性仍导致模型不鲁棒。为此,本文提出新的联合约束——变换稀疏约束与条件数约束(TSCNC),通过平滑分布与可微约束函数降低条件数,避免权重矩阵病态。理论分析揭示条件数与局部Lipschitz常数的相关性:条件数急剧上升成为过度稀疏模型鲁棒性的主要限制因素。在多个公开数据集上的实验表明,所提方法在高压缩率下显著提升DNN鲁棒性。
原文摘要 · Abstract (English)
Recent research has revealed that high compression of Deep Neural Networks (DNNs), e.g., massive pruning of the weight matrix of a DNN, leads to a severe drop in accuracy and susceptibility to adversarial attacks. Integration of network pruning into an adversarial training framework has been proposed to promote adversarial robustness. It has been observed that a highly pruned weight matrix tends to be ill-conditioned, i.e., increasing the condition number of the weight matrix. This phenomenon aggravates the vulnerability of a DNN to input noise. Although a highly pruned weight matrix is considered to be able to lower the upper bound of the local Lipschitz constant to tolerate large distortion, the ill-conditionedness of such a weight matrix results in a non-robust DNN model. To overcome this challenge, this work develops novel joint constraints to adjust the weight distribution of networks, namely, the Transformed Sparse Constraint joint with Condition Number Constraint (TSCNC), which copes with smoothing distribution and differentiable constraint functions to reduce condition number and thus avoid the ill-conditionedness of weight matrices. Furthermore, our theoretical analyses unveil the relevance between the condition number and the local Lipschitz constant of the weight matrix, namely, the sharply increasing condition number becomes the dominant factor that restricts the robustness of over-sparsified models. Extensive experiments are conducted on several public datasets, and the results show that the proposed constraints significantly improve the robustness of a DNN with high pruning rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。