给ViT加8个可学习的寄存器,让人脸识别更准且注意力更清晰
Vision Transformers for Face Recognition Need More Registers

- 引入可学习的寄存器令牌,与图像块共同处理,增强注意力结构
- 加8个寄存器后,在IJB-B/C上达到SOTA,准确率显著提升
- 注意力图更清晰可解释,适合需要模型透明性的应用场景
近年来,视觉变换器(ViTs)在人脸识别(FR)中已超越传统的分类令牌(CLS)范式。该范式通过在图像块嵌入前添加特殊分类令牌(CLS)来表示输入。另一种方法——拼接图像块嵌入(CPE)——则将所有图像块向量拼接为单一向量,并投影为紧凑的人脸表示。虽CPE表现优于基于CLS的方法,但我们的注意力图定性分析发现其存在干扰解释的伪影。为此,本文引入寄存器令牌(register tokens),即附加于初始图像块嵌入的可学习令牌,并与输入一起通过ViT编码器块联合处理。该机制生成了比基线更结构化、更可解释的注意力图。实验表明,此类伪影在多种ViT骨干网络(包括小型和大型模型)中普遍存在,而引入寄存器令牌能有效缓解问题。加入4或8个寄存器显著提升可解释性,其中8个寄存器取得最高验证准确率与最平滑的注意力结构。最终模型ViT-8R为基于CPE的ViT-B架构,增补8个寄存器,在大规模IJB-B与IJB-C基准上达到当前最优性能。同时,其注意力图明显更清晰,揭示了模型关注行为的深层机制。
原文摘要 · Abstract (English)
Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-token paradigm. In this paradigm, a special classification token (CLS) is prepended to the patch embeddings and used as a representation of the input for downstream tasks. An alternative approach, Concatenated Patch Embeddings (CPE), instead leverages all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation. CPE has been shown to improve recognition performance in comparison to CLS-based ones, but our qualitative analysis of attention maps showed the presence of artifacts that limit their interpretability. To address this issue, we incorporate register tokens, learnable tokens concatenated to the initial patch embeddings, and processed jointly through the ViT encoder blocks. This mechanism has been shown to produce more structured and interpretable attention maps compared to baseline ViT. We empirically demonstrate that these artifacts consistently appear across various ViT backbones, including small and large models, and that introducing register tokens effectively mitigates them. Adding four or eight registers significantly enhances interpretability, with eight registers providing the highest verification accuracies and smoothest attention structures. Our resulting model, ViT-8R, corresponds to a CPE-based ViT-B architecture augmented with eight register tokens achieves state-of-the-art performance among ViT-based FR models on large-scale IJB-B and IJB-C benchmarks. Also, ViT-8R produces substantially clearer attention maps compared with the baseline model, which offer deeper insight into the model's attention behavior (https://github.com/TaharChettaoui/ViT-FR-Registers)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。