arXiv:2410.05239cs.CVcs.CL2024-10中稿 · ACCV 2024被引 2

提出首个视觉-语言分割的提示调优基准,评估多种提示方法在医学图像上的表现。

TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models

  • 构建6种提示策略与2个模型组合,支持任意类别数的分割任务
  • 医学数据上文本提示效果差,视觉提示性能接近多模态且参数更少
  • 适合关注医疗图像分割、提示调优的开发者和研究者

视觉-语言模型在视觉任务中表现优异,但适应新领域常需昂贵微调。提示调优(包括文本、视觉和多模态提示)通过可学习提示提供高效替代方案。然而,其在视觉-语言分割模型(VLSMs)中的应用及在显著领域差异下的评估仍未知。本文提出开源基准框架TuneVLSeg,将多种单模态与多模态提示调优整合至VLSMs,使提示调优适用于任意类别数的下游分割数据集。TuneVLSeg包含6种提示策略在2个VLSM上的不同深度组合,共8种配置。我们在8个多样化医学数据集上测试,包括3个放射科数据集(乳腺肿瘤、超声心动图、胸部X光病变)和5个非放射科数据集(息肉、溃疡、皮肤癌),以及两个自然域分割数据集。结果发现,文本提示在从自然图像到医学数据的显著领域偏移下表现不佳;而视觉提示相比多模态提示参数更少,性能却常具竞争力,是值得优先尝试的方案。本工作推进了对不同提示调优技术在鲁棒领域特定分割中适用性的理解。代码已开源:https://github.com/naamiinepal/tunevlseg。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have shown impressive performance in vision tasks, but adapting them to new domains often requires expensive fine-tuning. Prompt tuning techniques, including textual, visual, and multimodal prompting, offer efficient alternatives by leveraging learnable prompts. However, their application to Vision-Language Segmentation Models (VLSMs) and evaluation under significant domain shifts remain unexplored. This work presents an open-source benchmarking framework, TuneVLSeg, to integrate various unimodal and multimodal prompt tuning techniques into VLSMs, making prompt tuning usable for downstream segmentation datasets with any number of classes. TuneVLSeg includes $6$ prompt tuning strategies on various prompt depths used in $2$ VLSMs totaling of $8$ different combinations. We test various prompt tuning on $8$ diverse medical datasets, including $3$ radiology datasets (breast tumor, echocardiograph, chest X-ray pathologies) and $5$ non-radiology datasets (polyp, ulcer, skin cancer), and two natural domain segmentation datasets. Our study found that textual prompt tuning struggles under significant domain shifts, from natural-domain images to medical data. Furthermore, visual prompt tuning, with fewer hyperparameters than multimodal prompt tuning, often achieves performance competitive to multimodal approaches, making it a valuable first attempt. Our work advances the understanding and applicability of different prompt-tuning techniques for robust domain-specific segmentation. The source code is available at https://github.com/naamiinepal/tunevlseg.

视觉语言分割提示调优医学图像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。