让视觉语言重排序更快更省，适合大规模应用。
Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking
- 预计算图像特征并用轻量适配器压缩，线上仅需小规模视觉令牌
- 每秒处理5万对图文数据，每图仅需49KB存储空间
- 在Flickr和COCO数据集上表现媲美现有最佳模型
多模态检索仍依赖CLIP等嵌入模型进行快速向量搜索，但与文本检索中常见的联合编码重排序器不同，视觉-语言重排序器仍不成熟。我们发现经典联合编码器如BLIP因昂贵的视觉特征提取阶段而严重受限，难以规模化部署。为此，我们提出EDJE——一种高效判别式联合编码器，通过离线预计算视觉令牌，并利用轻量级注意力适配器压缩，使在线推理仅需在少量视觉令牌与文本上运行紧凑的联合编码器。EDJE在大幅降低存储与在线计算开销的同时保持优异检索性能，实现每秒处理5万张图像-文本对，每图仅需49kB磁盘存储，零样本在Flickr、微调在COCO上的表现达到现有最优水平。
原文摘要 · Abstract (English)
Multimodal retrieval still leans on embedding-based models like CLIP for fast vector search over pre-computed image embeddings. Yet, unlike text retrieval, where joint-encoder rerankers are standard, comparable vision-language rerankers are largely absent. We find that seminal joint encoders such as BLIP are severely bottlenecked by an expensive visual feature-extraction stage, preventing practical deployment at scale. Motivated by this bottleneck, we introduce EDJE, an Efficient Discriminative Joint Encoder that precomputes vision tokens offline and compresses them via a lightweight attention-based adapter, so online inference runs only a compact joint encoder over a small set of visual tokens plus the text. EDJE preserves strong retrieval performance while drastically reducing storage and online compute, enabling high-throughput inference. Specifically, EDJE processes 50k image--text pairs/second while requiring 49kB of disk storage per image, matching prior art on Flickr (zero-shot) and COCO (fine-tuned) retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。