构建两个新数据集，用于测试图文混合检索的深度理解能力。
Entity Image and Mixed-Modal Image Retrieval Datasets
- 设计实体图像与多实体图文检索两类挑战性任务
- 基于WIT数据集构建，支持单/多实体图文查询
- 适合研究跨模态理解与检索的学者使用
尽管多模态学习取得进展，但结合视觉与文本信息的混合模态图像检索基准仍显不足。本文提出一个新基准，用于严格评估需要深层跨模态上下文理解的图像检索。我们构建了两个新数据集：实体图像数据集（EI），提供维基百科实体的标准图像；以及混合模态图像检索数据集（MMIR），源自WIT数据集。MMIR包含两种具有挑战性的查询类型：单实体-图像查询（一个实体图像配描述性文本）和多实体-图像查询（多个实体图像配关系型文本）。我们通过实证验证该基准在训练与评估混合模态检索中的有效性。两个数据集的质量通过众包人工标注进一步确认。数据集可在GitHub页面获取：https://github.com/google-research-datasets/wit-retrieval。
原文摘要 · Abstract (English)
Despite advances in multimodal learning, challenging benchmarks for mixed-modal image retrieval that combines visual and textual information are lacking. This paper introduces a novel benchmark to rigorously evaluate image retrieval that demands deep cross-modal contextual understanding. We present two new datasets: the Entity Image Dataset (EI), providing canonical images for Wikipedia entities, and the Mixed-Modal Image Retrieval Dataset (MMIR), derived from the WIT dataset. The MMIR benchmark features two challenging query types requiring models to ground textual descriptions in the context of provided visual entities: single entity-image queries (one entity image with descriptive text) and multi-entity-image queries (multiple entity images with relational text). We empirically validate the benchmark's utility as both a training corpus and an evaluation set for mixed-modal retrieval. The quality of both datasets is further affirmed through crowd-sourced human annotations. The datasets are accessible through the GitHub page: https://github.com/google-research-datasets/wit-retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。