通过图像内类别替换增强,实现无需教师模型的高效自蒸馏。
Intra-class Patch Swap for Self-Distillation
- 用同一模型内的同类图像块互换生成高低置信度样本对。
- 在分类、分割和检测任务上均超越传统蒸馏与现有自蒸馏方法。
- 无需额外参数或结构修改,适合快速部署到各类模型中。
知识蒸馏(KD)是将大型深度学习模型压缩为适合边缘设备的小型网络的重要技术。然而,传统KD框架依赖预训练的高容量教师模型,带来内存/存储开销大、训练成本高及教师选择模糊等问题。尽管无教师蒸馏(自蒸馏)成为有前景的替代方案,但多数方法仍需架构修改或复杂训练流程,限制了通用性与效率。为此,我们提出一种基于无教师蒸馏的新框架,仅使用单一学生网络,无需辅助组件、架构改动或可学习参数。该方法基于一种简单而高效的增强策略——类内图像块替换(intra-class patch swap),通过生成具有不同置信度的同类样本对,并进行实例间蒸馏以对齐预测分布,模拟教师-学生关系。本方法概念清晰、模型无关且易于实现,仅需一个增强函数。在图像分类、语义分割和目标检测任务上的大量实验表明,该方法持续优于现有自蒸馏基线和传统教师-学生蒸馏方法。结果表明,自蒸馏的成功可能取决于增强设计本身。代码已开源:https://github.com/hchoi71/Intra-class-Patch-Swap。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a valuable technique for compressing large deep learning models into smaller, edge-suitable networks. However, conventional KD frameworks rely on pre-trained high-capacity teacher networks, which introduce significant challenges such as increased memory/storage requirements, additional training costs, and ambiguity in selecting an appropriate teacher for a given student model. Although a teacher-free distillation (self-distillation) has emerged as a promising alternative, many existing approaches still rely on architectural modifications or complex training procedures, which limit their generality and efficiency. To address these limitations, we propose a novel framework based on teacher-free distillation that operates using a single student network without any auxiliary components, architectural modifications, or additional learnable parameters. Our approach is built on a simple yet highly effective augmentation, called intra-class patch swap augmentation. This augmentation simulates a teacher-student dynamic within a single model by generating pairs of intra-class samples with varying confidence levels, and then applying instance-to-instance distillation to align their predictive distributions. Our method is conceptually simple, model-agnostic, and easy to implement, requiring only a single augmentation function. Extensive experiments across image classification, semantic segmentation, and object detection show that our method consistently outperforms both existing self-distillation baselines and conventional teacher-based KD approaches. These results suggest that the success of self-distillation could hinge on the design of the augmentation itself. Our codes are available at https://github.com/hchoi71/Intra-class-Patch-Swap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。