用基础模型实现无需源数据的测试时自适应检测,突破类别封闭限制。
Test-Time Adaptive Object Detection with Foundation Model
- 基于多模态提示的Mean-Teacher框架,参数高效适配视觉与语言空间
- 在跨损坏与跨数据集测试中性能超越现有方法,支持任意领域与类别迁移
- 创新提出动态记忆模块,提升伪标签质量并支持无标签图像幻觉
近年来,测试时自适应目标检测因其在在线域自适应中的独特优势而受到关注,更贴近真实应用场景。然而,现有方法严重依赖源域统计特征,并假设源域与目标域具有相同的类别空间。本文提出首个基于基础模型的测试时自适应目标检测方法,完全消除对源数据的需求,突破传统闭集限制。我们设计了多模态提示驱动的Mean-Teacher框架,通过文本与视觉提示微调,参数高效地在测试数据上适配视觉与语言表示空间。同时,提出针对视觉提示的测试时预热策略,有效保留视觉分支表征能力。为确保每批次测试中伪标签高质量,我们引入实例动态记忆(IDM)模块,存储先前测试样本的优质伪标签,并提出两种新策略——记忆增强与记忆幻觉,分别用于提升原始预测和在无伪标签情况下生成图像。在跨损坏与跨数据集基准上的大量实验表明,本方法持续优于现有最先进方法,可适应任意跨域与跨类别目标数据。代码已开源:https://github.com/gaoyingjay/ttaod_foundation。
原文摘要 · Abstract (English)
In recent years, test-time adaptive object detection has attracted increasing attention due to its unique advantages in online domain adaptation, which aligns more closely with real-world application scenarios. However, existing approaches heavily rely on source-derived statistical characteristics while making the strong assumption that the source and target domains share an identical category space. In this paper, we propose the first foundation model-powered test-time adaptive object detection method that eliminates the need for source data entirely and overcomes traditional closed-set limitations. Specifically, we design a Multi-modal Prompt-based Mean-Teacher framework for vision-language detector-driven test-time adaptation, which incorporates text and visual prompt tuning to adapt both language and vision representation spaces on the test data in a parameter-efficient manner. Correspondingly, we propose a Test-time Warm-start strategy tailored for the visual prompts to effectively preserve the representation capability of the vision branch. Furthermore, to guarantee high-quality pseudo-labels in every test batch, we maintain an Instance Dynamic Memory (IDM) module that stores high-quality pseudo-labels from previous test samples, and propose two novel strategies-Memory Enhancement and Memory Hallucination-to leverage IDM's high-quality instances for enhancing original predictions and hallucinating images without available pseudo-labels, respectively. Extensive experiments on cross-corruption and cross-dataset benchmarks demonstrate that our method consistently outperforms previous state-of-the-art methods, and can adapt to arbitrary cross-domain and cross-category target data. Code is available at https://github.com/gaoyingjay/ttaod_foundation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。