简单知识蒸馏在语义分割中效果远超复杂新方法，关键看训练时长而非花招。
The Surprising Effectiveness of Canonical Knowledge Distillation for Semantic Segmentation
- 用基础的输出和特征蒸馏，不搞复杂设计
- 训练时间相同时，性能超越最新专用方法
- 小模型接近大模型，适合资源有限场景
近期语义分割的知识蒸馏方法引入越来越复杂的手工目标函数，但通常在固定迭代次数下评估。这些目标显著增加每轮开销，导致相同迭代数并不等同于相同计算预算。因此，报告的性能提升究竟是更强的蒸馏信号，还是更多算力所致尚不明确。我们发现基于迭代次数的比较具有误导性：当实际训练时间匹配时，传统的基于logit和特征的蒸馏方法优于近期提出的分割专用方法。在更长时间训练下，特征蒸馏实现了Cityscapes和ADE20K上ResNet-18的当前最优表现。一个PSPNet ResNet-18学生模型仅使用教师模型四分之一参数，却在Cityscapes上达到79.0的mIoU（教师为79.8），相当于99%；在ADE20K上达92%。结果挑战了‘分割蒸馏需任务特化机制’的普遍认知，表明未来设计应聚焦于模型规模扩展，而非复杂人工目标。
原文摘要 · Abstract (English)
Recent knowledge distillation (KD) methods for semantic segmentation introduce increasingly complex hand-crafted objectives, yet are typically evaluated under fixed iteration schedules. These objectives substantially increase per-iteration cost, meaning equal iteration counts do not correspond to equal training budgets. It is therefore unclear whether reported gains reflect stronger distillation signals or simply greater compute. We show that iteration-based comparisons are misleading: when wall-clock compute is matched, canonical logit- and feature-based KD outperform recent segmentation-specific methods. Under extended training, feature-based distillation achieves state-of-the-art ResNet-18 performance on Cityscapes and ADE20K. A PSPNet ResNet-18 student closely approaches its ResNet-101 teacher despite using only one quarter of the parameters, reaching 99% of the teacher's mIoU on Cityscapes (79.0 vs 79.8) and 92% on ADE20K. Our results challenge the prevailing assumption that KD for segmentation requires task-specific mechanisms and suggest that scaling, rather than complex hand-crafted objectives, should guide future method design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。