通过关系引导对抗学习,提升生成数据的类内多样性和类间混淆度。
Relation-Guided Adversarial Learning for Data-free Knowledge Transfer
- 设计双阶段优化:先生成类内分散、类间混淆的图像,再反向训练学生模型。
- 在多个基准上实现精度显著提升,数据效率更高。
- 适用于知识蒸馏、量化和增量学习等多种无数据场景。
数据自由知识蒸馏通过从预训练模型中恢复训练数据来传递知识。尽管近期研究关注全局数据多样性,但对类内多样性和类间相似性的忽视导致数据同质化,限制了性能。本文提出一种新型关系引导对抗学习方法(RGAL),结合三元组损失,从两个方面解决同质性问题:促进生成样本的类内多样性与类间混淆性。具体地,图像生成阶段通过优化使同标签样本远离、异标签样本靠近;学生训练阶段则反向操作,试图缩小同类别距离、扩大异类别距离。为缓解高全局多样性与类间混淆之间的冲突,提出聚焦加权采样策略,在有限距离范围内非均匀选择三元组负样本。RGAL在准确率和数据效率上显著优于现有最先进方法,并可嵌入多种数据自由知识迁移任务。实验在多个基准上验证了该方法的有效性与通用性,尤其在数据自由知识蒸馏、数据自由量化及非示例增量学习中表现优异。代码已开源。
原文摘要 · Abstract (English)
Data-free knowledge distillation transfers knowledge by recovering training data from a pre-trained model. Despite the recent success of seeking global data diversity, the diversity within each class and the similarity among different classes are largely overlooked, resulting in data homogeneity and limited performance. In this paper, we introduce a novel Relation-Guided Adversarial Learning method with triplet losses, which solves the homogeneity problem from two aspects. To be specific, our method aims to promote both intra-class diversity and inter-class confusion of the generated samples. To this end, we design two phases, an image synthesis phase and a student training phase. In the image synthesis phase, we construct an optimization process to push away samples with the same labels and pull close samples with different labels, leading to intra-class diversity and inter-class confusion, respectively. Then, in the student training phase, we perform an opposite optimization, which adversarially attempts to reduce the distance of samples of the same classes and enlarge the distance of samples of different classes. To mitigate the conflict of seeking high global diversity and keeping inter-class confusing, we propose a focal weighted sampling strategy by selecting the negative in the triplets unevenly within a finite range of distance. RGAL shows significant improvement over previous state-of-the-art methods in accuracy and data efficiency. Besides, RGAL can be inserted into state-of-the-art methods on various data-free knowledge transfer applications. Experiments on various benchmarks demonstrate the effectiveness and generalizability of our proposed method on various tasks, specially data-free knowledge distillation, data-free quantization, and non-exemplar incremental learning. Our code is available at https://github.com/Sharpiless/RGAL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。