arXiv:2506.22149cs.CV2025-06中稿 · presentation at MI…被引 3

用图文联合训练提升眼底影像模型语义理解,无需标注即可适配新人群。

RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models

  • 通过图文对比学习,用文本信息增强眼底图像模型的语义表征。
  • 在7个不同任务上平均提升5.8个百分点,显著改善下游分类性能。
  • 适用于医疗影像研究者,尤其适合资源有限场景下的模型迁移。

光学相干断层扫描(OCT)技术与深度学习的发展推动了视网膜疾病分期的自动化。自监督学习(SSL)使模型能从大量无标注数据中学习,催生了可应用于多种任务的基础模型(FMs)。然而,现有基于OCT图像的模型缺乏全面的语义理解能力,导致复杂任务表现不佳,通常需依赖昂贵的监督微调才能适配特定人群。为此,我们提出RetFiner,一种基于视觉-语言的自监督精炼方案,通过利用文本数据中的丰富监督信号,改进现有基础模型的表征能力,实现高效直接的群体适配。我们在RETFound、UrFound和VisionFM三个模型上测试,对七个高度多样化的OCT分类任务进行线性探测,平均准确率分别提升5.8、3.9和2.1个百分点。代码与模型权重已公开于https://github.com/ronnief1/RetFiner。

原文摘要 · Abstract (English)

The rise of imaging techniques such as optical coherence tomography (OCT) and advances in deep learning (DL) have enabled clinicians and researchers to streamline retinal disease staging. A popular DL approach is self-supervised learning (SSL), where models learn from vast amounts of unlabeled data, avoiding costly annotation. SSL has allowed the development of foundation models (FMs), large models that can be used for a variety of downstream tasks. However, existing FMs for OCT, trained solely on image data, lack a comprehensive and robust semantic understanding of images, as evidenced by their downstream performance (especially for complex tasks), and thus require supervised fine-tuning (which may be unfeasible) to better adapt to specific applications and populations. To address this, we propose RetFiner, an SSL vision-language refinement scheme that improves the representations of existing FMs and enables their efficient and direct adaptation to specific populations for improved downstream performance. Our method uses a diverse set of training objectives which take advantage of the rich supervisory signal found in textual data. We tested RetFiner on the retinal FMs RETFound, UrFound, and VisionFM, showing significant improvements in linear probing performance on seven highly diverse OCT classification tasks, with an average increase of 5.8, 3.9, and 2.1 percentage points over their baselines, respectively. Our code and model weights are publicly available at https://github.com/ronnief1/RetFiner.

眼底影像视觉语言自监督学习模型精炼

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。