arXiv:2512.08221cs.CV2025-12

构建视觉知识图谱,让机器理解动物的结构与关系。

VisKnow: Constructing Visual Knowledge Base for Object Understanding

  • 用图文对齐方法提取物体级多模态知识,构建成知识图谱。
  • 建成包含406类动物、22万条文本三元组的AnimalKB数据集。
  • 提升零样本识别与细粒度问答,适合视觉推理研究者使用。

物体理解是计算机视觉的基础。超越仅输出类别标签的识别任务,深层次理解需涵盖物体组成、外观特征、类别间关系及上下文知识。实现此目标依赖丰富的多模态数据,包括部件、属性、共现等视觉标注,以及支持推理和问答的文本知识。然而现有数据多为任务导向,缺乏系统组织。为此,我们提出视觉知识库(Visual Knowledge Base),以图结构组织多模态物体知识,并构建了名为VisKnow的构建框架,融合专家设计与大模型能力,整合文本与图像区域标注,实现物体与部件级别的知识提取。作为案例,我们构建了AnimalKB,覆盖406个动物类别,包含22,000条从百科文档中提取的文本三元组、420,000张图像及对应区域标注。实验表明,AnimalKB可显著提升零样本识别与细粒度视觉问答性能,并成为知识图谱补全与部件分割的挑战性基准。结果验证了自动构建视觉知识库在推动视觉理解及其应用方面的潜力。项目页面:https://vipl-vsu.github.io/VisKnow。

原文摘要 · Abstract (English)

Understanding objects is fundamental to computer vision. Beyond object recognition that provides only a category label as typical output, in-depth object understanding represents a comprehensive perception of an object category, involving its components, appearance characteristics, inter-category relationships, contextual background knowledge, etc. Developing such capability requires sufficient multi-modal data, including visual annotations such as parts, attributes, and co-occurrences for specific tasks, as well as textual knowledge to support high-level tasks like reasoning and question answering. However, these data are generally task-oriented and not systematically organized enough to achieve the expected understanding of object categories. In response, we propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow that extracts multi-modal, object-level knowledge for object understanding. This framework integrates enriched aligned text and image-source knowledge with region annotations at both object and part levels through a combination of expert design and large-scale model application. As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories, which contains 22K textual knowledge triplets extracted from encyclopedic documents, 420K images, and corresponding region annotations. A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA, and serves as challenging benchmarks for knowledge graph completion and part segmentation. Our findings highlight the potential of automatically constructing visual knowledge bases to advance visual understanding and its practical applications. The project page is available at https://vipl-vsu.github.io/VisKnow.

视觉理解知识图谱多模态动物识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。