视频测试时自适应新方法,显著提升极端分布偏移下的识别准确率。
Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts

- 通过双蒸馏机制动态调整视频模型,利用预训练视觉语言模型对齐特征
- 在三个基准上最高提升3.81%,优于现有最先进方法
- 仅更新轻量投影模块,适合实时视频场景部署
深度学习模型在多个计算机视觉任务中表现卓越,但在真实场景中因未预料的分布偏移导致性能严重下降。测试时自适应(TTA)通过使用目标域无标签数据,在推理时动态调整模型以适应测试分布,无需源数据。然而,当处理连续、时序相关的视频数据且目标域存在严重分布偏移时,现有方法仍面临挑战。为此,本文提出测试时自适应双蒸馏(TADD),一种基于冻结CLIP主干的在线适应框架,仅通过一个轻量级投影适配器实现更新。该适配器在源域预训练后,利用互补损失在目标域进行适应:(i) 零样本蒸馏,促使特征与预训练视觉语言模型的领域无关特征对齐;(ii) 目标蒸馏,保留源域判别性知识。在三个知名视频动作识别基准UCF-HMDB、Daily-DA和Sports-DA上评估,封闭集场景下性能持续超越现有最先进基线,分别提升+3.81%、+2.63%和+3.03%。
原文摘要 · Abstract (English)
Deep learning models have achieved state-of-the-art performance in several computer vision tasks. However, they experience severe performance degradation when applied to real-world scenarios due to unanticipated distribution shifts. Test-Time Adaptation (TTA) attempts to solve this problem by using unlabeled data from the target domain to dynamically adapt to the test distribution at inference time, without access to the source data. However, TTA remains a challenging problem when adapting to continuous, temporally correlated data, such as videos, and in scenarios where the target domain contains severe domain shifts. For this reason, few works in the literature explore TTA for videos under such extreme conditions. To overcome these limitations, we propose Test-time Adaptation via Dual Distillation (TADD), an online adaptation framework that relies on a lightweight projection adapter to bridge the domain gap. The adapter module is pre-trained on the source domain and then adapted to the target using our proposed complementary losses: (i) zero-shot distillation, which encourages alignment with the domain-agnostic features from a pre-trained vision-language model (VLM); and (ii) target distillation, which retains the source domain discriminative knowledge encoded in the pre-trained adapter. Built upon a frozen CLIP backbone, our method introduces this lightweight projection adapter as the sole updatable component during inference. We conducted extensive evaluations on three well-known video action recognition benchmarks: UCF-HMDB, Daily-DA, and Sports-DA. Our experiments in the closed-set scenario demonstrate that our method consistently outperforms state-of-the-art TTA baselines. Notably, our TTA approach improves upon previous methods by up to +3.81% on UCF-HMDB, +2.63% on Daily-DA, and +3.03% on Sports-DA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。