通过去除图文冗余信息，提升视觉语言模型的细粒度对齐效果。
Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models
- 双向精炼：移除高重叠图像块和相似文本描述
- 在6个数据集上显著提升零样本性能
- 适合改进CLIP类模型的图文对齐能力
近期研究表明，将细粒度文本描述与局部图像区域对齐可显著提升预训练视觉语言模型（如CLIP）的零样本性能。然而我们发现，细粒度文本描述与局部图像区域常包含冗余信息，导致图文对齐效果下降。本文从两个角度解决该问题：视图精炼（View Refinement）和描述精炼（Description Refinement），统称为BiFTA（Bi-refinement for Fine-grained Text-visual Alignment）。视图精炼通过移除交并比（IoU）高的图像块，获得更具区分性的视觉样本；描述精炼则移除成对余弦相似度高的文本描述，确保剩余描述具有更高多样性。BiFTA在基于ViT和ResNet的CLIP模型上，于6个基准数据集均取得更优的零样本性能，验证了在视觉-文本对齐中去除冗余信息的必要性。
原文摘要 · Abstract (English)
Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: \emph{View Refinement} and \emph{Description refinement}, termed as \textit{\textbf{Bi}-refinement for \textbf{F}ine-grained \textbf{T}ext-visual \textbf{A}lignment} (BiFTA). \emph{View refinement} removes redundant image patches with high \emph{Intersection over Union} (IoU) ratios, resulting in more distinctive visual samples. \emph{Description refinement} removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。