提出统一自集成框架,提升测试时提示调优的稳定性与效果
USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

- 通过自集成策略,让模型更关注原图而非增强图像以生成可靠伪标签
- 在多个数据集上性能超越现有方法,提升幅度达1.8%-4.2%
- 既可独立使用,也可无缝集成到其他测试时适配方法中
测试时适应(TTA)已成为提升视觉语言模型(如CLIP)在下游任务表现的主流范式。在现有基于CLIP的TTA方法中,测试时提示调优(TPT)是开创性工作,通过多组测试时增强优化文本提示,至今仍是强基线。本文重新审视TPT,发现其优化过程可被解释为隐式利用自生成伪标签。基于此视角,我们提出统一自集成框架(USE),确保优化与推理阶段的一致性。优化阶段引入简单有效的自集成(SE)策略,自适应地强化测试图像本身而非其增强视图,以获得更可靠的伪标签。为进一步挖掘增强的潜力,我们在推理阶段也应用相同策略,统一两个阶段的目标。值得注意的是,SE本身也可作为轻量级无优化的TTA方法。在多个数据集上的大量实验表明,SE与USE分别优于对应方法。此外,将SE整合进现有TTA方法能持续带来性能提升。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。