将模糊图像转为文本描述，提升多模态知识图谱的补全效果。
Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment
- 通过三阶段流程：大规模图像检索、图像转文本、多源文本融合生成实体摘要。
- 在含模糊图标的数据集上，Hits@1提升333.33%，MRR提升201.35%。
- 适合关注多模态知识图谱、视觉语义理解的研究者与开发者。
多模态知识图谱（MMKG）受益于视觉信息，但大规模图像收集困难，且常遗漏模糊但相关的视觉内容（如标识、符号、抽象画面）。我们提出 Beyond Images，一种自动化的数据驱动增强流程，支持人工审核。该流程分三步：(1) 大规模检索与实体相关的额外图像；(2) 将所有视觉输入转换为文本描述，使模糊图像贡献可用语义而非噪声；(3) 使用大语言模型（LLM）融合多源描述，生成简洁、对齐实体的摘要。这些摘要可直接替换或补充标准 MMKG 模型中的文本模态，无需修改模型架构或损失函数。在三个公开的 MMKG 数据集和多个基线模型上，均实现一致提升（最高达7% Hits@1）。尤其在包含模糊标识与符号的挑战子集上，图像转文本带来显著改进（MRR提升201.35%，Hits@1提升333.33%）。此外，我们发布轻量级文本-图像一致性检查接口，支持针对性人工审计，提升描述质量与数据可靠性。结果表明，扩大图像覆盖并转换模糊视觉内容为文本，是增强 MMKG 补全能力的可行路径。代码、数据集及补充材料见 https://github.com/pengyu-zhang/Beyond-Images。
原文摘要 · Abstract (English)
Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。