用视觉语言模型提升眼底图像无源域适应,防遗忘且关注病灶细节。
Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model
- 设计防遗忘模块保留目标模型自信预测,避免性能下降。
- 引入病灶感知模块,利用ViL模型的细粒度信息提升诊断精度。
- 适合医学图像分析、无源域适应研究者参考。
无源域适应(SFDA)旨在仅使用未标注的目标域数据和源模型的情况下,将源域训练好的模型适配到目标域。现有方法在域偏移下易出错,近年开始借助现成的视觉语言(ViL)模型。但当前工作存在两方面问题:(i) 尽管利用互信息融合ViL与目标模型的预测,仍出现目标模型部分类别预测能力退化;(ii) 忽视了ViL模型中蕴含的丰富细粒度知识,尤其在眼底图像诊断中。本文提出一种新型的防遗忘且病灶感知(FRLA)方法,通过防遗忘模块显式保留目标模型的高置信预测,病灶感知模块从ViL模型获取局部块级预测,帮助目标模型识别病灶区域并利用其细粒度知识。大量实验表明,该方法不仅显著优于单纯使用ViL模型,还持续超越当前最优方法。代码将公开。
原文摘要 · Abstract (English)
Source-free domain adaptation (SFDA) aims to adapt a model trained in the source domain to perform well in the target domain, with only unlabeled target domain data and the source model. Taking into account that conventional SFDA methods are inevitably error-prone under domain shift, recently greater attention has been directed to SFDA assisted with off-the-shelf foundation models, e.g., vision-language (ViL) models. However, existing works of leveraging ViL models for SFDA confront two issues: (i) Although mutual information is exploited to consider the joint distribution between the predictions of ViL model and the target model, we argue that the forgetting of some superior predictions of the target model still occurs, as indicated by the decline of the accuracies of certain classes during adaptation; (ii) Prior research disregards the rich, fine-grained knowledge embedded in the ViL model, which offers detailed grounding for fundus image diagnosis. In this paper, we introduce a novel forgetting-resistant and lesion-aware (FRLA) method for SFDA of fundus image diagnosis with ViL model. Specifically, a forgetting-resistant adaptation module explicitly preserves the confident predictions of the target model, and a lesion-aware adaptation module yields patch-wise predictions from ViL model and employs them to help the target model be aware of the lesion areas and leverage the ViL model's fine-grained knowledge. Extensive experiments show that our method not only significantly outperforms the vision-language model, but also achieves consistent improvements over the state-of-the-art methods. Our code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。