arXiv:2603.02554cs.CV2026-03中稿 · CVPR被引 1

让小模型学会大模型的泛化能力,尤其在未见过的数据上表现更好。

Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation

  • 分两阶段训练:先学通用特征,再固定特征学任务,避免过拟合
  • 用查询机制从大模型中挑选可迁移的空间知识,提升跨域适应力
  • 在5个测试集上平均提升10.6%,特别适合从大模型迁移到小模型

知识蒸馏(KD)广泛用于语义分割中的大模型压缩,但传统方法主要关注域内准确率,忽视域外泛化能力,这在分布偏移下至关重要。随着视觉基础模型(VFMs)的出现,尽管其在未见数据上表现出强鲁棒性,但使用传统KD蒸馏会损害这种能力。本文提出通用知识蒸馏(GKD),一种多阶段框架,显式增强泛化性能。GKD将表示学习与任务学习解耦:第一阶段,学生模型通过选择性特征蒸馏获取领域无关表示;第二阶段,冻结这些表示以进行任务适配,从而缓解对可见领域的过拟合。为进一步支持迁移,引入基于查询的软蒸馏机制,使学生特征作为查询,从教师表示中选择性检索可迁移的空间知识。在五个域泛化基准上的大量实验表明,GKD持续优于现有方法,在基础到基础(F2F)蒸馏中平均提升+1.9%,在基础到本地(F2L)蒸馏中提升+10.6%。代码将发布于https://github.com/Younger-hua/GKD。

原文摘要 · Abstract (English)

Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at https://github.com/Younger-hua/GKD.

知识蒸馏语义分割域泛化视觉模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。