按深度动态增加载荷，用少244倍参数实现医疗影像分割高效微调
Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging
- 根据层深与语义相关性，逐层递增适配器容量
- 仅61.3万参数即在5个医学数据集上超越传统微调
- 适合资源受限临床场景的轻量级模型部署
将视觉语言分割模型（VLSMs）适配至医学影像领域时，传统微调方法计算开销巨大。现有参数高效微调（PEFT）方法对所有变压器层采用统一适配器尺寸，导致参数分配不均、适应效率低下。本文提出新型PEFT框架——望远镜适配器（Telescopic Adapters），通过深度感知缩放机制，使适配器容量从浅层到深层逐步增强。该方法在CLIPSeg的视觉与文本编码器中集成轻量瓶颈模块，适配器尺寸依据层深和语义相关性动态调整。仅使用61.3万可训练参数（较全量微调减少244倍），在涵盖息肉分割、皮肤病变检测与乳腺超声成像的五个多样化医学数据集上均取得更优性能。全面消融实验表明，深层网络需远高于浅层的适配能力，验证了望远镜缩放假设。本方法为医疗VLSM高效微调树立新范式，支持在资源受限临床环境中部署，同时保持高精度分割能力。代码已公开于https://github.com/Ujjwal238/Telescopic_adapters。
原文摘要 · Abstract (English)
Adapting Vision Language Segmentation Models (VLSMs) to medical imaging domains requires significant computational overhead when using conventional fine-tuning approaches. Existing Parameter-Efficient Fine-Tuning (PEFT) methods apply uniform adapter dimensions across all transformer layers, leading to suboptimal parameter allocation and reduced adaptation efficiency. We introduce Telescopic Adapters, a novel PEFT framework that employs depth-aware scaling to progressively increase adapter capacity from shallow to deep transformer layers. Our method integrates lightweight bottleneck modules within CLIPSeg's vision and text encoders, with adapter dimensions dynamically scaled based on layer depth and semantic relevance. Using only 613k trainable parameters--244x fewer than end-to-end fine-tuning, Telescopic Adapters achieve superior performance across five diverse medical datasets spanning polyp segmentation, skin lesion detection, and breast ultrasound imaging. Comprehensive ablation studies demonstrate that deeper layers require substantially more adaptation capacity than shallow layers, validating our telescopic scaling hypothesis. Our approach establishes a new paradigm for efficient medical VLSM fine-tuning, enabling deployment in resource-constrained clinical environments while maintaining competitive segmentation accuracy. Our source code is publicly available at https://github.com/Ujjwal238/Telescopic_adapters
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。