通过抑制冗余信息，实现跨架构知识蒸馏的高效简化。
Cross-Architecture Distillation Made Simple with Redundancy Suppression
- 将知识迁移转化为冗余信息抑制问题，避免复杂设计。
- 在CIFAR-100和ImageNet-1k上超越OFA，参数开销极低。
- 无需架构特异性设计，适合快速部署与基准对比。
我们提出一种简单的跨架构知识蒸馏方法，将知识迁移建模为冗余信息抑制问题。现有方法依赖复杂模块、定制化设计和大量参数，影响效率与通用性。本文通过减少架构专属信息，提取异构表示中的通用知识，设计了仅包含跨架构不变性最大化与特征去相关目标的轻量级冗余抑制蒸馏（RSD）损失。为防止学生模型丧失自身架构特性，还引入一个轻量级解耦模块，使RSD目标与其内部表示分离。该方法不依赖特定架构设计或复杂操作，相比OFA方法在CIFAR-100和ImageNet-1k上表现更优，且参数开销仅为后者的一小部分，展现出作为跨架构蒸馏新基准的巨大潜力。
原文摘要 · Abstract (English)
We describe a simple method for cross-architecture knowledge distillation, where the knowledge transfer is cast into a redundant information suppression formulation. Existing methods introduce sophisticated modules, architecture-tailored designs, and excessive parameters, which impair their efficiency and applicability. We propose to extract the architecture-agnostic knowledge in heterogeneous representations by reducing the redundant architecture-exclusive information. To this end, we present a simple redundancy suppression distillation (RSD) loss, which comprises cross-architecture invariance maximisation and feature decorrelation objectives. To prevent the student from entirely losing its architecture-specific capabilities, we further design a lightweight module that decouples the RSD objective from the student's internal representations. Our method is devoid of the architecture-specific designs and complex operations in the pioneering method of OFA. It outperforms OFA on CIFAR-100 and ImageNet-1k benchmarks with only a fraction of their parameter overhead, which highlights its potential as a simple and strong baseline to the cross-architecture distillation community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。