arXiv:2605.04447cs.CV2026-05

提出DRD框架,让医疗大模型更高效地适配具体任务。

Deep Reprogramming Distillation for Medical Foundation Models

论文配图:Deep Reprogramming Distillation for Medical Foundation Models
图 1 · 摘自论文原文
  • 引入重编程模块,解决预训练与下游任务的差异问题。
  • 在18个医疗任务上超越现有方法,实现轻量化高效迁移。
  • 适合需要快速部署、资源受限的医疗场景应用。

医疗基础模型在大规模数据上预训练后展现出强大的通用性能。然而,在将这些模型适配到具体医疗场景时,仍面临因预训练与下游任务间差距、实际计算与速度限制带来的挑战。现有技术如知识蒸馏(KD)要求教师与学生模型共享相同任务、训练策略和模型结构族,而主流参数高效微调(PEFT)难以实现个性化且轻量化的部署。即使结合两者,仍无法解决教师与学生模型在结构和训练策略上的不一致,导致知识迁移效率低下。本文提出一种新框架——深度重编程蒸馏(DRD),通过创新的重编程模块,一方面缓解预训练与下游任务间的领域和任务差异,另一方面构建从基础模型到轻量级下游模型的友好蒸馏路径。此外,为减少不同训练条件下的波动性,设计了中心化核对齐(CKA)蒸馏方法以增强知识迁移的鲁棒性。实验表明,DRD在18个不同医疗下游任务中均优于先前的PEFT与KD方法,涵盖2D/3D分类与2D/3D分割等多种场景。

原文摘要 · Abstract (English)

Medical foundation models pre-trained on large-scale datasets have shown powerful versatile performance. However, when adapting medical foundation models for specific medical scenarios, it remains the inevitable challenge due to the gap induced by the discrepancy between pre-training and downstream tasks, the real-world computation, and speed constraints. Relevant techniques that probably handle this challenge more or less suffer from some intrinsic limitations. For example, knowledge distillation (KD) assumes that teacher and student models share the same task, training strategy, and model structure family, while prevalent parameter-efficient fine-tuning (PEFT) fails to achieve personalized and lightweight deployment. Even the combination of PEFT and KD still struggles to resolve model structures and training strategies inconsistencies between teacher and student models, leading to inefficient knowledge transfer. In this study, we propose a novel framework called Deep Reprogramming Distillation (DRD) to combat the general adaptation challenge. Specifically, DRD introduces the novel reprogramming module that on the one side overcomes the domain and task discrepancy between pretraining and downstream scenarios, and on the other side builds the student-friendly efficient distillation from foundation models to lightweight downstream models. Furthermore, to mitigate variability under different training conditions, we design a centered kernel alignment (CKA) distillation method to promote robust knowledge transfer. Empirical results show that DRD surpasses previous PEFT and KD methods across 18 medical downstream tasks under different foundation models, covering various scenarios including 2D/3D classification and 2D/3D segmentation.

医疗AI知识蒸馏轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。