提出轻量级卷积适配器，显著降低图像分割模型训练显存占用。
CAD: Memory Efficient Convolutional Adapter for Segment Anything
- 并行连接图像编码器，无需存储激活值和梯度
- 显存使用不足SAM适配器的一半，性能相当
- 适合显存受限场景下的模型微调
图像分割基础模型Segment Anything（SAM）自提出以来在多个领域受到广泛关注。为将其适配到特定任务，研究者常引入轻量级适配模块进行微调。尽管此类方法具备参数高效与性能提升优势，却存在一个被忽视的问题：训练时显存消耗远高于可训练参数数量。为此，本文提出一种内存高效的并行卷积适配器架构，该架构与SAM图像编码器并行连接，训练时无需保存图像编码器的激活值和梯度。实验表明，该方法在保持竞争力性能的同时，所需显存低于SAM适配器的一半，是硬件受限条件下替代简单解码器微调的可行方案。代码已开源。
原文摘要 · Abstract (English)
The Foundation model for image segmentation, Segment Anything (SAM), has been actively researched in various fields since its proposal. Various researches have been proposed to adapt SAM to specific domains, with one notable approach involving the addition and training of lightweight adapter modules. While adapter-based fine-tuning approaches have reported parameter efficiency and significant performance improvements, they face a often overlooked issue: the excessive consumption of GPU memory relative to the number of trainable parameters. Addressing this issue, this paper proposes a memory-efficient parallel convolutional adapter architecture. This architecture connects in parallel with SAM's image encoder, eliminating the need to store activations and gradients of the image encoder during model training. Our proposed architecture demonstrated competitive experimental results while using less than half the GPU memory compared to SAM Adapter, indicating its value as an alternative to simple decoder fine-tuning when hardware limitations preclude adapter-based learning. Our code implementation is available at our github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。