用大模型自动构建高质量网页级图像实体识别数据集
Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
- 用多模态大模型推理候选实体标签，结合维基百科等上下文提升准确性
- 自动生成问答对和细粒度解释文本，丰富数据语义信息
- 数据训练的模型在OVEN任务上提升6.9%，验证了高质量数据的重要性
网页级视觉实体识别旨在将图像与维基百科等大规模知识库中的对应实体关联，但因缺乏干净的大规模训练数据而面临挑战。本文提出一种新方法，利用多模态大语言模型（LLM）进行标签验证、元数据生成和理由解释，而非直接依赖其标注数据。我们通过让模型访问维基百科等上下文信息，推理潜在候选实体标签，从而获得更准确的标注。此外，利用多模态大模型生成问题-答案对和基于图像的细粒度文本描述（称为“理由”），以增强数据语义。实验表明，基于该自动构建数据训练的模型在网页级视觉实体识别任务中达到当前最优性能，如OVEN实体任务提升6.9%，凸显高质量训练数据在此领域的重要性。
原文摘要 · Abstract (English)
Web-scale visual entity recognition, the task of associating images with their corresponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data. In this paper, we propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Instead of relying on the multimodal LLM to directly annotate data, which we found to be suboptimal, we prompt it to reason about potential candidate entity labels by accessing additional contextually relevant information (such as Wikipedia), resulting in more accurate annotations. We further use the multimodal LLM to enrich the dataset by generating question-answer pairs and a grounded finegrained textual description (referred to as "rationale") that explains the connection between images and their assigned entities. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks (e.g. +6.9% improvement in OVEN entity task), underscoring the importance of high-quality training data in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。