让视觉模型在测试时自动适应新分布，提升泛化能力
Test-Time Visual In-Context Tuning
- 用单个测试样本反向优化提示，实现测试时快速适应
- 在15种图像损坏下，显著提升模型对未知分布的鲁棒性
- 适用于未见过的任务，适合部署在动态环境中的视觉系统
视觉上下文学习（VICL）作为一种新兴范式，使模型仅需少量提示和示例即可快速适应各类任务。然而，现有方法在分布偏移下泛化能力差。本文提出测试时视觉上下文微调（VICT），可在测试时通过单个样本动态调整模型。核心思想是反转任务提示与测试样本的角色，利用循环一致性损失重构原始提示输出。若模型能成功恢复原始提示，说明其已感知新分布。在六类代表性视觉任务（从高层理解到低层图像处理）及15种常见图像损坏下的实验表明，VICT显著提升了VICL在未见域上的泛化性能。此外，还展示了VICT在测试时应对未见任务的潜力。
原文摘要 · Abstract (English)
Visual in-context learning (VICL), as a new paradigm in computer vision, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. While effective, the existing VICL paradigm exhibits poor generalizability under distribution shifts. In this work, we propose test-time Visual In-Context Tuning (VICT), a method that can adapt VICL models on the fly with a single test sample. Specifically, we flip the role between the task prompts and the test sample and use a cycle consistency loss to reconstruct the original task prompt output. Our key insight is that a model should be aware of a new test distribution if it can successfully recover the original task prompts. Extensive experiments on six representative vision tasks ranging from high-level visual understanding to low-level image processing, with 15 common corruptions, demonstrate that our VICT can improve the generalizability of VICL to unseen new domains. In addition, we show the potential of applying VICT for unseen tasks at test time. Code: https://github.com/Jiahao000/VICT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。