arXiv:2410.01573cs.CV2024-10被引 14

提出PASS框架,让医学图像分割在测试时自动适应风格与形状差异。

PASS:Test-Time Prompting to Adapt Styles and Semantic Shapes in Medical Image Segmentation

  • 用输入感知的视觉提示调整图像风格,匹配预训练模型
  • 通过跨注意力机制生成目标特异的形状提示,提升分割精度
  • 无需源数据或训练阶段修改,适合真实医疗场景部署

测试时适应(TTA)为处理不同机构医学图像的领域偏移提供了新路径,无需额外训练数据。现有方法或依赖源域训练信息,或忽视分割任务中关键的形状语义知识。本文揭示图像风格和形状变异是导致性能下降的双重因素。为此提出PASS框架,联合学习两类提示:输入空间提示用于重构测试图像风格以适配预训练模型,语义感知提示用于弥合跨域高阶形状差异。通过输入装饰器生成自调节视觉提示,并设计跨注意力提示调制器,实现目标特征与丰富形状提示库的交互。大量实验表明,PASS在多个医学图像分割数据集上优于现有最优方法。代码已开源。

原文摘要 · Abstract (English)

Test-time adaptation (TTA) has emerged as a promising paradigm to handle the domain shifts at test time for medical images from different institutions without using extra training data. However, existing TTA solutions for segmentation tasks suffer from (1) dependency on modifying the source training stage and access to source priors or (2) lack of emphasis on shape-related semantic knowledge that is crucial for segmentation tasks.Recent research on visual prompt learning achieves source-relaxed adaptation by extended parameter space but still neglects the full utilization of semantic features, thus motivating our work on knowledge-enriched deep prompt learning. Beyond the general concern of image style shifts, we reveal that shape variability is another crucial factor causing the performance drop. To address this issue, we propose a TTA framework called PASS (Prompting to Adapt Styles and Semantic shapes), which jointly learns two types of prompts: the input-space prompt to reformulate the style of the test image to fit into the pretrained model and the semantic-aware prompts to bridge high-level shape discrepancy across domains. Instead of naively imposing a fixed prompt, we introduce an input decorator to generate the self-regulating visual prompt conditioned on the input data. To retrieve the knowledge representations and customize target-specific shape prompts for each test sample, we propose a cross-attention prompt modulator, which performs interaction between target representations and an enriched shape prompt bank. Extensive experiments demonstrate the superior performance of PASS over state-of-the-art methods on multiple medical image segmentation datasets. The code is available at https://github.com/EndoluminalSurgicalVision-IMR/PASS.

医学图像测试时适应视觉提示分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。