通过注意力引导的文本掩码与增强，提升跨模态对齐效果。
Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search
- 用注意力权重动态选择要掩码的关键词，避免无意义词干扰
- 在三个数据集上达到78.36%、67.31%、67.4%的Rank-1准确率
- 适合关注文本描述质量与跨模态对齐的研究者
在文本驱动的人体搜索（TBPS）中，主流方法致力于优化文本与视觉信息间的交互。然而，现有方法面临两大挑战：一是基于随机掩码的掩码语言建模（MLM）对所有词语同等处理，导致大量无语义词（如‘with’、‘the’）被掩码，阻碍跨模态表示对齐；二是人工标注的文本描述冗余且含误差。为此，本文提出注意力引导对齐（AGA）框架，包含两个创新模块：注意力引导掩码（AGM）和文本增强模块（TEM）。AGM通过聚合文本编码过程中的注意力权重，动态掩码语义重要词，使跨模态MLM能从上下文与图像中捕捉掩码词信息，实现更优对齐。TEM则通过用MLM预测替换语义关键词，缓解低质量描述带来的表征偏差，同时防止过拟合。在三个基准数据集上的实验证明，AGA取得新最优结果，CUHK-PEDES、ICFG-PEDES和RSTPReid上的Rank-1准确率分别为78.36%、67.31%和67.4%。
原文摘要 · Abstract (English)
In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data. However, recent approaches encounter two principal challenges. Firstly, the widely used random-based Masked Language Modeling (MLM) considers all the words in the text equally during training. However, massive semantically vacuous words ('with', 'the', etc.) be masked fail to contribute efficient interaction in the cross-modal MLM and hampers the representation alignment. Secondly, manual descriptions in TBPS datasets are tedious and inevitably contain several inaccuracies. To address these issues, we introduce an Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM). AGM dynamically masks semantically meaningful words by aggregating the attention weight derived from the text encoding process, thereby cross-modal MLM can capture information related to the masked word from text context and images and align their representations. Meanwhile, TEM alleviates low-quality representations caused by repetitive and erroneous text descriptions by replacing those semantically meaningful words with MLM's prediction. It not only enriches text descriptions but also prevents overfitting. Extensive experiments across three challenging benchmarks demonstrate the effectiveness of our AGA, achieving new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。