用概率掩码同时实现模型压缩与跨域泛化,无需额外训练。
Domain-Aware Pruning: Sparsity and Domain Generalization via Regularized Probabilistic Masking
- 学习连续保留概率,动态筛选领域无关的网络权重。
- 在五个基准数据集上实现显著稀疏性,且泛化性能不降反升。
- 适合追求高效、鲁棒且可解释模型的研究者和工程应用。
领域泛化(DG)与神经网络剪枝传统上被视为两个独立目标,分别关注分布外(OOD)鲁棒性和模型效率。本文提出领域感知剪枝(DAP),通过网络稀疏性作为提升未见领域泛化能力的机制,弥合这一鸿沟。不同于标准的二值掩码优化,DAP学习一个连续的参数保留概率 $p \in [0, 1]$,将网络压缩建模为连续的概率掩码问题。通过引入正则化目标,主动惩罚训练中对领域敏感权重的保留,从而识别出领域不变子网络。在五个DG基准数据集上的实验证明,DAP在实现显著稀疏性的同时,其分布外性能始终匹配或超越对应稠密模型。关键的是,DAP是算法无关框架,可无缝集成至现有DG流程,无需后续微调。此外,DAP天然提升对抗扰动鲁棒性,并生成高度可解释的模型,保留的权重可靠地编码最领域不变且任务关键的表征。
原文摘要 · Abstract (English)
Domain generalization (DG) and neural network pruning are conventionally treated as distinct objectives, targeting out-of-distribution (OOD) robustness and model efficiency, respectively. In this work, we bridge this gap by introducing Domain-Aware Pruning (DAP), a framework that leverages network sparsity as a mechanism to implicitly enhance generalization to unseen domains. Diverging from standard binary mask optimization, DAP learns a continuous parameter retention probability $p \in [0, 1]$, framing network compression as a continuous probabilistic masking problem. By introducing a regularization objective that actively penalizes the retention of domain-sensitive weights during the mask training, DAP identifies a domain-invariant subnetwork. Empirical results across five DG benchmark datasets demonstrate that DAP achieves significant sparsity while consistently matching or exceeding the OOD performance of its dense counterparts. Crucially, DAP is an algorithm-agnostic framework that integrates seamlessly with existing DG pipelines without necessitating post-hoc fine-tuning. Beyond efficiency and generalization, we show that DAP natively provides increased robustness to adversarial perturbations and yields highly interpretable models, where the retained weights reliably encapsulate the most domain-invariant and task-critical representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。