针对医疗影像跨域测试时自适应中碎片化数据流,提出图像级解耦提示调优方法。
F^2TTA: Free-Form Test-Time Adaptation on Cross-Domain Medical Image Classification via Image-Level Disentangled Prompt Tuning
- 采用图像不变与图像特异性双提示机制,应对未知域偏移。
- 在乳腺癌和青光眼分类上显著优于现有TTA方法。
- 适合临床中不规则数据流场景,尤其适用于资源受限环境。
测试时自适应(TTA)因可利用无标注测试数据降低标注成本,成为适应源模型到未见医疗站点的有力方案。现有方法假设数据以完整域单元形式到达,但实际临床中数据常以任意长度片段、随机顺序到来。本文研究一种更贴近实践的自由形式测试时自适应(F²TTA)任务,其中域偏移在片段间不可预测,易干扰适应过程。为此,提出图像级解耦提示调优(I-DiPT)框架:利用图像不变提示挖掘域不变表征以缓解偏移,图像特定提示则逐图适配。由于仅单图训练,提示知识表达不足,因此引入不确定性导向掩码(UoM),通过源模型表示的不确定性驱动掩码一致性学习,增强信息提取;进一步提出并行图蒸馏(PGD),通过并行图网络复用历史图像特定与不变提示的知识。在乳腺癌与青光眼分类任务上的实验表明,该方法在F²TTA设置下显著优于现有TTA方法。代码已开源。
原文摘要 · Abstract (English)
Test-Time Adaptation (TTA) has emerged as a promising solution for adapting a source model to unseen medical sites using unlabeled test data, due to the high cost of data annotation. Existing TTA methods consider scenarios where data from one or multiple domains arrives in complete domain units. However, in clinical practice, data usually arrives in domain fragments of arbitrary lengths and in random arrival orders, due to resource constraints and patient variability. This paper investigates a practical Free-Form Test-Time Adaptation (F$^{2}$TTA) task, where a source model is adapted to such free-form domain fragments, with shifts occurring between fragments unpredictably. In this setting, these shifts could distort the adaptation process. To address this problem, we propose a novel Image-level Disentangled Prompt Tuning (I-DiPT) framework. I-DiPT employs an image-invariant prompt to explore domain-invariant representations for mitigating the unpredictable shifts, and an image-specific prompt to adapt the source model to each test image from the incoming fragments. The prompts may suffer from insufficient knowledge representation since only one image is available for training. To overcome this limitation, we first introduce Uncertainty-oriented Masking (UoM), which encourages the prompts to extract sufficient information from the incoming image via masked consistency learning driven by the uncertainty of the source model representations. Then, we further propose a Parallel Graph Distillation (PGD) method that reuses knowledge from historical image-specific and image-invariant prompts through parallel graph networks. Experiments on breast cancer and glaucoma classification demonstrate the superiority of our method over existing TTA approaches in F$^{2}$TTA. Code is available at https://github.com/mar-cry/F2TTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。