用低秩适配让视觉语言模型测试时自动优化，提升泛化能力。
LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
- 仅微调图像编码器的低秩参数，测试时动态调整。
- 在15个数据集上平均提升5.79%零样本准确率。
- 无需外部模型或缓存，适合资源受限场景部署。
视觉语言模型（如CLIP）的快速发展加剧了训练与测试数据分布不一致的问题。现有测试时训练（TTT）方法多依赖文本提示调优，需大量计算资源且严重依赖熵损失。本文提出LoRA-TTT，将低秩适配（LoRA）应用于视觉语言模型的图像编码器，在测试时仅更新其参数，保持原始模型泛化能力的同时，实现显著性能提升，且内存与运行开销极小。此外，设计了一种高效重构损失，结合两种损失可适应多样领域，不增加内存或运行时间。在两个基准上覆盖15个数据集的实验表明，该方法使CLIP-ViT-B/16在域外（OOD）基准上平均零样本准确率提升5.79%，细粒度基准提升1.36%，显著优于测试时提示调优，且无需外部模型或缓存。
原文摘要 · Abstract (English)
The rapid advancements in vision-language models (VLMs), such as CLIP, have intensified the need to address distribution shifts between training and testing datasets. Although prior Test-Time Training (TTT) techniques for VLMs have demonstrated robust performance, they predominantly rely on tuning text prompts, a process that demands substantial computational resources and is heavily dependent on entropy-based loss. In this paper, we propose LoRA-TTT, a novel TTT method that leverages Low-Rank Adaptation (LoRA), applied exclusively to the image encoder of VLMs. By introducing LoRA and updating only its parameters during test time, our method offers a simple yet effective TTT approach, retaining the model's initial generalization capability while achieving substantial performance gains with minimal memory and runtime overhead. Additionally, we introduce a highly efficient reconstruction loss tailored for TTT. Our method can adapt to diverse domains by combining these two losses, without increasing memory consumption or runtime. Extensive experiments on two benchmarks, covering 15 datasets, demonstrate that our method improves the zero-shot top-1 accuracy of CLIP-ViT-B/16 by an average of 5.79% on the OOD benchmark and 1.36% on the fine-grained benchmark, efficiently surpassing test-time prompt tuning, without relying on any external models or cache.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。