让视觉语言模型在测试时自动识别并适应新类别和新领域。
Segmentation Assisted Incremental Test Time Adaptation in an Open World
- 用分割能力辅助主动采样，无须训练即可发现潜在新类别
- 在多个数据集上提升模型对未知类别的识别准确率
- 适合需要持续学习的开放世界应用场景
在动态环境中，模型常遇到未见过的物体和分布变化，挑战其泛化能力。本文针对视觉语言模型的增量测试时自适应（ITTA）问题，处理测试阶段持续出现未知类别与未知域的场景。不同于传统测试时自适应仅限于预定义类别，本框架可同时应对协变量与标签分布偏移，并在检测到新类别时主动适应。为此，我们构建了新的ITTA基准，融合单图自适应方法与主动标注技术，通过查询专家（oracle）获取可能代表未知类别的样本。提出无需训练的分割辅助主动标注模块SegAssist，利用视觉语言模型的分割能力优化样本选择，优先筛选出可能属于新类别的样本。大量实验表明，SegAssist显著提升了视觉语言模型在真实开放世界场景下的持续适应性能。项目页：https://manogna-s.github.io/segassist/
原文摘要 · Abstract (English)
In dynamic environments, unfamiliar objects and distribution shifts are often encountered, which challenge the generalization abilities of the deployed trained models. This work addresses Incremental Test Time Adaptation of Vision Language Models, tackling scenarios where unseen classes and unseen domains continuously appear during testing. Unlike traditional Test Time Adaptation approaches, where the test stream comes only from a predefined set of classes, our framework allows models to adapt simultaneously to both covariate and label shifts, actively incorporating new classes as they emerge. Towards this goal, we establish a new benchmark for ITTA, integrating single image TTA methods for VLMs with active labeling techniques that query an oracle for samples potentially representing unseen classes during test time. We propose a segmentation assisted active labeling module, termed SegAssist, which is training free and repurposes the segmentation capabilities of VLMs to refine active sample selection, prioritizing samples likely to belong to unseen classes. Extensive experiments on several benchmark datasets demonstrate the potential of SegAssist to enhance the performance of VLMs in real world scenarios, where continuous adaptation to emerging data is essential. Project-page:https://manogna-s.github.io/segassist/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。