无需训练即可提升多模态检索准确率的简单修正方法
Nearest Neighbor Normalization Improves Multimodal Retrieval
- 基于最近邻统计对对比模型输出进行归一化修正
- 在多个模型和数据集上均显著提升图文检索指标
- 适合希望提升现有模型性能的研究者使用
多模态模型通过大规模预训练在图像描述、视觉问答和跨模态检索等任务上取得了强劲但仍有不足的表现。本文提出一种无需额外训练的简单高效方法——最近邻归一化（Nearest Neighbor Normalization, NNN），用于修正已训练的对比图像-文本检索模型中的错误。我们在所有测试的对比模型（CLIP、BLIP、ALBEF、SigLIP、BEiT）和两个数据集（MS-COCO、Flickr30k）上均观察到文本检索与图像检索指标的提升。NNN需依赖参考数据库，但无需在该库上进行训练，甚至可在微调后进一步提升模型检索精度。
原文摘要 · Abstract (English)
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。