arXiv:2607.15919cs.LGcs.AI2026-07

边界寻找蒸馏在生成模型中失效,因特征间耦合导致梯度冲突。

On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures

论文配图:On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
图 1 · 摘自论文原文
  • 将重建任务转为特征级分类,暴露生成模型的边界缺陷。
  • 在MNIST上实验表明边界采样引发严重梯度冲突。
  • 提出流形感知合成,适合作为无数据生成蒸馏基线。

无数据知识蒸馏可在不访问原始数据的情况下将教师模型的知识迁移到学生模型。先前工作如对比归纳知识提取(CAKE)通过生成靠近教师决策边界的样本,实现了分类器的蒸馏。本文研究该边界寻找策略是否适用于自编码器蒸馏,在MNIST数据集上进行实验。为实现直接对比,我们将连续重建重构为每特征的密集分类任务,使解码器输出类别逻辑值。结果表明,边界寻找目标在瓶颈化生成架构中本质上是病态的:CAKE基于单实例级目标运作,但解码器实质是受共享低维瓶颈约束的紧密耦合特征级分类器阵列。独立为这些耦合输出采样对比目标会破坏学习到的潜在流形几何结构,导致严重梯度冲突,而非产生有信息量的边界样本。流形感知合成则完全规避此类冲突,建立起无数据生成蒸馏的有效基线。

原文摘要 · Abstract (English)

Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) achieves this for classifiers by synthesizing samples near the teacher's decision boundary. In this work, we investigate whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset . To enable a direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We show that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, but a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts instead of informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely and establishes an effective baseline for data-free generative distillation.

知识蒸馏生成模型边界采样流形学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。