提出新方法加速视觉变压器模型逆向,提升生成质量与迁移能力。
Patch Rebirth: Toward Fast and Transferable Model Inversion of Vision Transformers
- 通过渐进式保留关键图像块,动态重建合成输入
- 速度比标准方法快10倍,比现有优化方法快2倍
- 适合需要高效数据重构的隐私研究与模型分析
模型逆向是数据无关学习中常用的技术,通过迭代优化从预训练模型中重建合成输入,无需原始训练数据。然而,将其应用于先进的视觉变换器(ViTs)面临巨大计算挑战,因其自注意力机制开销高。为应对这一问题,稀疏模型逆向(SMI)通过剪枝并丢弃看似不重要的图像块来提升效率,甚至认为这些块会阻碍知识迁移。但我们的实证发现相反:即使随机选择的图像块也能通过持续逆向获得可迁移知识。这表明过早丢弃图像块效率低下,会抑制对类无关特征和类特定特征的提取。本文提出一种新方法——图像块重生逆向(PRI),在逆向过程中逐步分离最重要图像块以构建稀疏合成图像,同时让其余图像块持续演化以备未来选择。该渐进策略不仅提升效率,还促使初始信息量少的图像块逐步积累更多类别相关知识,我们称之为‘重生效应’,有效平衡了类无关与类特定知识。实验表明,PRI比标准密集模型逆向(DMI)快10倍,比SMI快2倍,且在准确率上持续优于SMI,性能接近DMI。
原文摘要 · Abstract (English)
Model inversion is a widely adopted technique in data-free learning that reconstructs synthetic inputs from a pretrained model through iterative optimization, without access to original training data. Unfortunately, its application to state-of-the-art Vision Transformers (ViTs) poses a major computational challenge, due to their expensive self-attention mechanisms. To address this, Sparse Model Inversion (SMI) was proposed to improve efficiency by pruning and discarding seemingly unimportant patches, which were even claimed to be obstacles to knowledge transfer. However, our empirical findings suggest the opposite: even randomly selected patches can eventually acquire transferable knowledge through continued inversion. This reveals that discarding any prematurely inverted patches is inefficient, as it suppresses the extraction of class-agnostic features essential for knowledge transfer, along with class-specific features. In this paper, we propose Patch Rebirth Inversion (PRI), a novel approach that incrementally detaches the most important patches during the inversion process to construct sparse synthetic images, while allowing the remaining patches to continue evolving for future selection. This progressive strategy not only improves efficiency, but also encourages initially less informative patches to gradually accumulate more class-relevant knowledge, a phenomenon we refer to as the Re-Birth effect, thereby effectively balancing class-agnostic and class-specific knowledge. Experimental results show that PRI achieves up to 10x faster inversion than standard Dense Model Inversion (DMI) and 2x faster than SMI, while consistently outperforming SMI in accuracy and matching the performance of DMI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。