用模型自身注意力引导细粒度对齐，提升文本搜人准确率
ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval
- 利用模型自注意力生成高显著性标记池，实现无监督局部对齐
- 在三个主流数据集上达到最新最好性能，跨数据集泛化能力强
- 适合关注视觉语言对齐与细粒度检索的研究者
视觉语言模型在基于文本的人体搜索（TBPS）任务中表现优异，该任务需捕捉图像与文本间的细粒度关联以区分个体。现有方法依赖局部对齐，但易受捷径学习和虚假相关影响，导致错位；注入先验知识还会破坏模态内结构。我们发现编码器注意力从训练初期即能提供空间精确的证据，据此提出ITSELF框架，通过引导表示与注意力银行（GRAB）将模型自身注意力转化为高显著性标记池，并在该池上施加局部目标，实现无需额外监督的细粒度对应学习。为确保选择可靠且非冗余，引入多层注意力鲁棒选择（MARS），聚合多层注意力并进行多样性感知的top-k选择；以及自适应标记调度器（ATS），在训练过程中从粗到细逐步调整保留预算，早期保留上下文，后期聚焦判别性细节。在三个广泛使用的TBPS基准上的大量实验表明，本方法达到当前最优性能，并展现出强跨数据集泛化能力，验证了方法的有效性与鲁棒性，且无需额外先验监督。项目代码公开于 https://trhuuloc.github.io/itself
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Moreover, injecting prior knowledge can distort intra-modality structure. Motivated by our finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, and to alleviate these issues, we introduceITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. To make the selection reliable and non-redundant, we introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection; and Adaptive Token Scheduler (ATS), which schedules the retention budget from coarse to fine over training, preserving context early while progressively focusing on discriminative details. Extensive experiments on three widely used TBPS benchmarks showstate-of-the-art performance and strong cross-dataset generalization, confirming the effectiveness and robustness of our approach without additional prior supervision. Our project is publicly available at https://trhuuloc.github.io/itself
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。