针对在线域适应中注意力特征漂移问题,提出渐进式重校准方法提升模型鲁棒性。
Progressive Conditioned Scale-Shift Recalibration of Self-Attention for Online Test-time Adaptation
- 分层渐进式重校准自注意力,通过条件缩放与偏移参数动态调整
- 在ImageNet-C上实现最高3.9%的分类准确率提升
- 轻量级网络在线适配,适合部署于实时推理场景
在线测试时适应旨在推理阶段根据连续输入样本动态调整模型。本文发现,将Transformer模型应用于新目标域时,其自注意力模块的查询、键和值特征相比源域显著变化,导致性能大幅下降。为此,提出一种新方法:通过由条件缩放和偏移因子参数化的局部线性变换,逐层渐进重校准自注意力。将从源域到目标域的在线适应视为渐进域偏移分离过程,在每一层学习域分离网络提取域偏移特征,并通过因子生成网络预测自注意力重校准所需的缩放与偏移参数。这两个轻量网络在推理过程中在线适配。在基准数据集上的实验表明,所提渐进式条件缩放-偏移重校准(PCSR)方法在ImageNet-C数据集上分类准确率最高提升达3.9%。
原文摘要 · Abstract (English)
Online test-time adaptation aims to dynamically adjust a network model in real-time based on sequential input samples during the inference stage. In this work, we find that, when applying a transformer network model to a new target domain, the Query, Key, and Value features of its self-attention module often change significantly from those in the source domain, leading to substantial performance degradation of the transformer model. To address this important issue, we propose to develop a new approach to progressively recalibrate the self-attention at each layer using a local linear transform parameterized by conditioned scale and shift factors. We consider the online model adaptation from the source domain to the target domain as a progressive domain shift separation process. At each transformer network layer, we learn a Domain Separation Network to extract the domain shift feature, which is used to predict the scale and shift parameters for self-attention recalibration using a Factor Generator Network. These two lightweight networks are adapted online during inference. Experimental results on benchmark datasets demonstrate that the proposed progressive conditioned scale-shift recalibration (PCSR) method is able to significantly improve the online test-time domain adaptation performance by a large margin of up to 3.9\% in classification accuracy on the ImageNet-C dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。