用教师模型软标签蒸馏，让小模型性能超越自身极限
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles
- 用多个教师模型生成软标签，指导学生模型训练
- 在ICHBI数据集上达64.39新高，比前人提升0.85
- 无需改架构，适配各种模型，尤其适合资源受限场景
呼吸音数据集规模小、质量差，导致高性能难以实现。集成模型虽能提升效果，但推理时计算开销大。软标签训练可在仅增加训练成本的前提下高效传递知识。本文探索将软标签用于呼吸音分类，提出一种与架构无关的知识蒸馏方法，将多个教师模型的知识蒸馏到一个学生模型中。我们测试了不同变体，发现即使单个教师与学生结构相同，也能显著提升性能；使用少数教师即可获得最优增益。在ICHBI数据集上达到64.39的新最优得分，较之前最佳提升0.85，并在各类架构上平均提升超过1.16。结果表明，无论模型大小或结构如何，基于软标签的知识蒸馏在呼吸音分类中均十分有效。
原文摘要 · Abstract (English)
Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。