用链式结构让小模型一步步学大模型,少标签也能高效精准。
WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging
- 构建多级串联模型链,每层从前一层提炼知识并优化传递。
- 仅用部分数据训练,累积准确率提升最高达23%。
- 适合标注数据少的医疗影像场景,实用性强。
知识蒸馏(KD)传统上依赖静态师生框架,大型教师模型向单个学生模型传递知识,但常面临知识退化、监督效率低及对强教师模型或大规模标注数据的依赖。为此,我们提出首个弱监督链式知识蒸馏网络(WeCKD),通过一系列相互连接的模型实现渐进式知识传递。每个模型不仅从前驱模型学习,还对知识进行精炼后再向前传递,显著增强特征学习并克服单步蒸馏的局限。链中每个模型仅在数据集的一小部分上训练,证明了在极小监督下仍可实现有效学习。在六种跨耳镜、显微镜和磁共振成像模态的影像数据集上的广泛评估显示,该方法具有优异泛化能力,优于现有方法。此外,该蒸馏链在相同有限数据条件下,相比单一骨干网络,累积准确率最高提升23%,展现出其在真实医疗场景中的应用潜力。
原文摘要 · Abstract (English)
Knowledge distillation (KD) has traditionally relied on a static teacher-student framework, where a large, well-trained teacher transfers knowledge to a single student model. However, these approaches often suffer from knowledge degradation, inefficient supervision, and reliance on either a very strong teacher model or large labeled datasets. To address these, we present the first-ever Weakly-supervised Chain-based KD network (WeCKD) that redefines knowledge transfer through a structured sequence of interconnected models. Unlike conventional KD, it forms a progressive distillation chain, where each model not only learns from its predecessor but also refines the knowledge before passing it forward. This structured knowledge transfer further enhances feature learning and addresses the limitations of one-step KD. Each model in the chain is trained on only a fraction of the dataset and shows that effective learning can be achieved with minimal supervision. Extensive evaluation on six imaging datasets across otoscopic, microscopic, and magnetic resonance imaging modalities shows that it generalizes and outperforms existing methods. Furthermore, the proposed distillation chain resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data, which highlights its potential for real-world adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。