用轻量视觉模型实时生成掩码,无反向传播就能精准适应分布偏移。
TestMate: Test-Time Domain Adaptation Aided by Lightweight Vision Foundation Model

- 基于轻量视觉模型生成多尺度无标注掩码,实时融合主模型预测
- 首帧即生效,小目标边界保留好,准确率在两个数据集上达新高
- 无需参数更新,适合部署在资源受限的实时系统中
测试时域自适应(TTDA)旨在仅使用实时流式、未标注测试数据,对深度神经网络进行分布偏移适应。现有语义分割方法存在严重局限:熵最小化需代价高昂的反向传播,易导致灾难性遗忘并产生噪声分割边界;记忆库方法虽免于反向传播,但适应缓慢,需大量样本才能收敛,难以应对连续域偏移。本文提出TestMate,一种新颖的实时、无反向传播的TTDA框架,克服上述问题。TestMate利用轻量级视觉基础模型的泛化能力引导适应过程。采用零样本实例分割YOLOv8-seg模型,实时生成多尺度对象及其部分的未标注掩码提议。这些提议通过启发式大小排序竞争机制与主模型融合,小而高置信度区域主导并精修周围大而低置信度区域的预测。该无参机制实现首帧即适应,天然避免灾难性遗忘,有效保留细粒度物体细节和边界,即使对小物体亦然。TestMate可作为独立高效精炼模块,或无缝集成至现有TTDA方法以显著提升性能。我们在两个基准数据集上展示其在三种不同适应任务(TTDA、源无关域自适应SFDA、在线TTDA)中的领先表现。代码已开源。
原文摘要 · Abstract (English)
Test-Time Domain Adaptation (TTDA) aims to adapt Deep Neural Networks to distribution shifts using only streaming, unlabeled test data in real time. Current methods for semantic segmentation tasks suffer from critical limitations. Entropy minimization techniques require costly backpropagation, risking catastrophic forgetting and producing noisy segmentation boundaries. Memory-bank methods, while backpropagation-free, exhibit slow adaptation, requiring numerous samples to converge and struggle to handle continuous domain shifts. We introduce TestMate, a novel, real-time, and backpropagation-free TTDA framework that overcomes these issues. TestMate leverages generalization capability of a lightweight Visual Foundation Model to guide the adaptation. We use a zero-shot instance segmentation YOLOv8-seg based model to generate unlabeled mask proposals for objects and their parts at multiple scales in real time. These proposals are fused with the primary model via a heuristic, size-ordered competitive scheme, where small, high-confidence regions dominate and refine predictions in surrounding larger, less certain areas. This paremeter-free mechanism enables immediate adaptation from the first frame, inherently avoids catastrophic forgetting and effectively preserves fine object details and boundaries, even for small objects. TestMate can be used as a standalone, efficient refinement module or seamlessly integrated into existing TTDA methods to significantly boost their performance. We demonstrate state-of-the-art results across two benchmark datasets, proving TestMate's effectiveness in three distinct adaptation tasks: TTDA, Source-Free Domain Adaptation (SFDA), and online-TTDA. Code is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。