用对比学习提升图文实体识别效率，速度比生成模型快100倍
WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition
- 用大语言模型嵌入+视觉引导的知识适配器对齐图文特征
- 在OVEN未见集上提升16%准确率，推理延迟降低近100倍
- 适合需要快速部署的开放域图文匹配场景
开放域视觉实体识别（VER）旨在将图像与维基百科等百科知识库中的实体关联。近期针对VER的生成方法虽表现优异，但计算开销高，限制了可扩展性与实际部署。本文重新审视对比学习范式，提出WikiCLIP——一个简单而高效的框架，为开放域VER建立了强而高效的基准。WikiCLIP利用大语言模型嵌入作为富含知识的实体表示，并通过视觉引导的知识适配器（VGKA）在图像块级别对齐文本语义与视觉线索。为进一步增强细粒度区分能力，引入硬负样本合成机制，在训练中生成视觉相似但语义不同的负样本。在OVEN等主流开放域VER基准上的实验表明，WikiCLIP显著优于强基线。具体而言，在挑战性的OVEN未见集上提升16%，同时相比领先生成模型AutoVER，推理延迟降低近100倍。
原文摘要 · Abstract (English)
Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large language model embeddings as knowledge-rich entity representations and enhances them with a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16\% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/project_pages/WikiCLIP/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。