用CLIP模型实现图文混合搜索海量地图库,支持自然语言和图像查询。
Integrating Visual and Textual Inputs for Searching Large-Scale Map Collections with CLIP
- 基于CLIP模型对56万张地图生成多模态嵌入,支持文本与图像联合搜索。
- 构建10,504对地图-描述数据集,提升地图语义理解能力。
- 开源全部代码与交互式笔记本,助力文博机构开展智能检索。
尽管地图在数字馆藏中普遍存在且具有重要历史价值,但当前导航和探索地图库的方法仍主要依赖目录记录和结构化元数据。本文探索通过自然语言输入(如“有海怪的地图”)、视觉输入(反向图像搜索)及多模态输入(一张示例图+“更多灰度”)来交互式搜索大规模地图库的潜力。以美国国会图书馆开放API提供的562,842张地图图像为例,我们使用多模态对比语言-图像预训练(CLIP)模型生成地图嵌入,并开发代码实现多种查询方式。实验结果基于国会图书馆地理与地图部工作人员的咨询设计,展示了各类查询的优缺点及应用前景。此外,我们提出包含10,504组地图-标题配对的微调数据集,并提供在该数据集上微调CLIP模型的架构。为促进复用,所有代码均以文档化、可交互的Jupyter笔记本形式公开,置于公共领域。最后,讨论了这些方法在美术馆、图书馆、档案馆和博物馆所持的数字化及原生数字馆藏中的应用机遇与挑战。
原文摘要 · Abstract (English)
Despite the prevalence and historical importance of maps in digital collections, current methods of navigating and exploring map collections are largely restricted to catalog records and structured metadata. In this paper, we explore the potential for interactively searching large-scale map collections using natural language inputs ("maps with sea monsters"), visual inputs (i.e., reverse image search), and multimodal inputs (an example map + "more grayscale"). As a case study, we adopt 562,842 images of maps publicly accessible via the Library of Congress's API. To accomplish this, we use the mulitmodal Contrastive Language-Image Pre-training (CLIP) machine learning model to generate embeddings for these maps, and we develop code to implement exploratory search capabilities with these input strategies. We present results for example searches created in consultation with staff in the Library of Congress's Geography and Map Division and describe the strengths, weaknesses, and possibilities for these search queries. Moreover, we introduce a fine-tuning dataset of 10,504 map-caption pairs, along with an architecture for fine-tuning a CLIP model on this dataset. To facilitate re-use, we provide all of our code in documented, interactive Jupyter notebooks and place all code into the public domain. Lastly, we discuss the opportunities and challenges for applying these approaches across both digitized and born-digital collections held by galleries, libraries, archives, and museums.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。