arXiv:2608.09287cs.CVcs.AI2026-08

提出统一语义先验框架,让模型压缩无需原始数据且适配各类网络结构。

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

论文配图:UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation
图 1 · 摘自论文原文
  • 用语言嵌入和高斯先验替代依赖网络结构的统计信息生成数据
  • 在CNN与ViT上均实现超20%性能提升,跨架构效果显著
  • 适合无原始数据、需压缩复杂模型的场景

数据自由知识蒸馏(DFKD)通过合成语义信息丰富的数据,将预训练教师模型的知识迁移至小型学生模型,无需访问原始训练数据。现有方法严重依赖特定网络结构的统计先验(如批量归一化统计量),但在现代网络(如视觉变换器ViTs)中常缺失此类信息,导致合成数据语义质量下降,性能严重退化。本文提出 extit{UniDFKD},一种统一的无数据知识蒸馏框架,以显式的、与架构无关的语义先验替代原有架构依赖的统计量。该框架从三个维度控制整个合成-蒸馏流程:(1) 类别语义条件(CSC)通过持续使用语言导出的嵌入调节生成器,捕捉语义多样性;(2) 空间语义锚定(SSA)通过将教师模型的空间注意力图锚定到高斯先验,明确特征证据的位置;(3) 空间语义蒸馏(SSD)通过显式对齐教师与学生模型的空间证据及预测结果,控制知识传递方式。在多种CNN与ViT架构上的大量实验表明,UniDFKD达到新标杆,在同构与异构设置下平均绝对性能超越现有方法超过20%。

原文摘要 · Abstract (English)

Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose \emph{UniDFKD}, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. \emph{UniDFKD} governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines \emph{what} to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates \emph{where} evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls \emph{how} knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20\% in both homogeneous and heterogeneous settings.

知识蒸馏无数据训练视觉变换器语义先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。