用CLIP理解博物馆展品，能自动补全表格信息
Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits
- 用Transformer解析网络将图像嵌入转为表格结构
- 在20万+图文对上实现高精度属性值补全
- 适合文物数字化、智能导览等场景
CLIP虽强大，但缺乏针对特定任务的细粒度与结构化理解能力。本文提出MUZE方法，旨在实现博物馆展品的细粒度、表格化视觉理解。我们构建并公开首个面向该任务的20万+图像-表格对数据集，提出基于Transformer的解析网络parseNet，将CLIP图像嵌入映射为表格结构。该方法能利用已知属性对上下文预测缺失属性值，显著提升准确率。在新建立的基准测试中，模型表现优异。代码与数据集已开源。
原文摘要 · Abstract (English)
CLIP is a powerful and widely used tool for understanding images in the context of natural language descriptions to perform nuanced tasks. However, it does not offer application-specific fine-grained and structured understanding, due to its generic nature. In this work, we aim to adapt CLIP for fine-grained and structured -- in the form of tabular data -- visual understanding of museum exhibits. To facilitate such understanding we (a) collect, curate, and benchmark a dataset of 200K+ image-table pairs, and (b) develop a method that allows predicting tabular outputs for input images. Our dataset is the first of its kind in the public domain. At the same time, the proposed method is novel in leveraging CLIP's powerful representations for fine-grained and tabular understanding. The proposed method (MUZE) learns to map CLIP's image embeddings to the tabular structure by means of a proposed transformer-based parsing network (parseNet). More specifically, parseNet enables prediction of missing attribute values while integrating context from known attribute-value pairs for an input image. We show that this leads to significant improvement in accuracy. Through exhaustive experiments, we show the effectiveness of the proposed method on fine-grained and structured understanding of museum exhibits, by achieving encouraging results in a newly established benchmark. Our dataset and source-code can be found at: https://github.com/insait-institute/MUZE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。