arXiv:2410.10442cs.CV2024-10被引 9

通过动态调整注意力机制,让模型在测试时自动适应新领域。

Domain-Conditioned Transformer for Fully Test-time Adaptation

  • 在Transformer注意力中引入领域条件向量,动态调节特征表示。
  • 在多个测试域上实现显著性能提升,优于现有最先进方法。
  • 适合需要在线适应新环境的部署场景,如跨域图像识别。

完全测试时自适应旨在推理阶段基于输入样本的序列分析在线调整网络模型。我们观察到,将Transformer模型应用于新领域时,目标域图像样本的自注意力分布与源域存在显著差异,导致领域切换时性能大幅下降。为解决此问题,我们提出一种新的自注意力模块结构:在查询、键和值组件中引入三个领域条件向量(称为领域调节器),并设计网络从每层Transformer的类别标记生成这些向量。实验表明,在完全在线测试时自适应过程中,各层的领域调节器能够逐步消除领域偏移影响,显著恢复原始自注意力模式。大量实验证明,所提领域条件化Transformer显著提升了在线自适应性能,大幅超越现有最先进方法。

原文摘要 · Abstract (English)

Fully test-time adaptation aims to adapt a network model online based on sequential analysis of input samples during the inference stage. We observe that, when applying a transformer network model into a new domain, the self-attention profiles of image samples in the target domain deviate significantly from those in the source domain, which results in large performance degradation during domain changes. To address this important issue, we propose a new structure for the self-attention modules in the transformer. Specifically, we incorporate three domain-conditioning vectors, called domain conditioners, into the query, key, and value components of the self-attention module. We learn a network to generate these three domain conditioners from the class token at each transformer network layer. We find that, during fully online test-time adaptation, these domain conditioners at each transform network layer are able to gradually remove the impact of domain shift and largely recover the original self-attention profile. Our extensive experimental results demonstrate that the proposed domain-conditioned transformer significantly improves the online fully test-time domain adaptation performance and outperforms existing state-of-the-art methods by large margins.

自适应Transformer测试时学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。