通过几何关系引导蒸馏,提升医学视觉语言模型在少样本下的准确率与稳定性。
Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models

- 引入类间几何结构生成方向性目标,避免忽略临床相关类别关系。
- 在11个医学数据集上平均提升1.7%-2.8%准确率,显著优于现有方法。
- 适合医疗图像少样本微调,尤其对未见类别泛化能力强。
当前基于提示和适配器的视觉语言模型微调方法在医学影像中备受关注,因其可避免冻结主干网络并适应标注稀缺场景。然而,这些方法通常仅优化真实类别,将所有其他类别视为同等错误,忽视了临床有意义的类别关系,导致有限监督下决策边界不稳定。本文提出全向几何知识蒸馏(OGKD),通过在教师模型中注入类别关系结构,生成既保留真实标签又尊重类间几何的方向性目标。基于此目标,设计两种蒸馏损失:全局几何感知蒸馏(GAD)作用于全局图像标记,标签引导几何蒸馏(LGD)则应用于注意力局部标记以增强细粒度对齐。在11个常用医学数据集上的基线到新类别及少样本评估中,OGKD性能显著优于现有最先进方法,平均绝对提升1.7%-2.8%。其在未见类别上也展现出更强鲁棒性和更可靠的预测能力。代码已开源:https://github.com/tientrandinh/OGKD。
原文摘要 · Abstract (English)
Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging, where clinical data sensitivity favors frozen backbones and annotations are limited. However, these methods typically optimize only the ground-truth class, treating all other classes as equally incorrect, ignoring clinically meaningful class relations and yielding unstable decision boundaries in limited-supervision settings. We propose Omni-Geometry Knowledge Distillation (OGKD), a new framework that injects class-relation structure into the teacher to produce directional targets that preserve the ground truth while respecting inter-class geometry. Using these targets, we develop two distillation losses: Global Geometry-Aware Distillation (GAD) operates on the global image token, and Label-Guided Geometry Distillation (LGD) applies the same geometry to attentive patch tokens to improve fine-grained alignment. Across comprehensive experiments and analyses on 11 widely-used medical datasets for base-to-novel and few-shot evaluations, our OGKD achieves substantially better performance, consistently improving accuracy by an average absolute gain of 1.7%-2.8% over all prior state-of-the-art VLM adaptation counterparts. It also robustly generalizes to unseen classes and yields more reliable predictions than other approaches. Our code is available at https://github.com/tientrandinh/OGKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。