将参考图像对视为伪视频，让SAM2无需重训练就能高效适应新场景。
Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild
- 把参考图与目标图的对应关系建模为伪视频序列
- 在多个数据集上性能超越现有方法5%以上
- 适合需要快速适配新任务的轻量级应用
大型视觉模型如分割一切模型（SAM）在真实场景下游任务中表现受限。参考分割通过参考图像及其掩码向模型注入新知识，成为新方向。但现有方法多依赖元学习，需大量训练数据与计算资源。本文提出将参考-目标图像对的内在对应关系表示为伪视频，利用具备交互式视频目标分割能力的SAM2，实现轻量级测试时自适应。所提方法名为CAV-SAM，包含两个模块：基于扩散的语义过渡（DBST）构建语义变换序列，测试时几何对齐（TTGA）通过测试时微调校正序列中的几何变化。在多个主流数据集上验证，分割性能超越当前最优方法超过5%。代码与补充材料已提供。
原文摘要 · Abstract (English)
Large vision models like the Segment Anything Model (SAM) exhibit significant limitations when applied to downstream tasks in the wild. Consequently, reference segmentation, which leverages reference images and their corresponding masks to impart novel knowledge to the model, emerges as a promising new direction for adapting vision models. However, existing reference segmentation approaches predominantly rely on meta-learning, which still necessitates an extensive meta-training process and brings massive data and computational cost. In this study, we propose a novel approach by representing the inherent correspondence between reference-target image pairs as a pseudo video. This perspective allows the latest version of SAM, known as SAM2, which is equipped with interactive video object segmentation (iVOS) capabilities, to be adapted to downstream tasks in a lightweight manner. We term this approach Correspondence As Video for SAM (CAV-SAM). CAV-SAM comprises two key modules: the Diffusion-Based Semantic Transition (DBST) module employs a diffusion model to construct a semantic transformation sequence, while the Test-Time Geometric Alignment (TTGA) module aligns the geometric changes within this sequence through test-time fine-tuning. We evaluated CAVSAM on widely-used datasets, achieving segmentation performance improvements exceeding 5% over SOTA methods. Implementation is provided in the supplementary materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。