arXiv:2605.10120cs.CVcs.AI2026-05

用知识图谱增强大模型,让其在显微镜图像理解上表现大幅提升。

MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph

论文配图:MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph
图 1 · 摘自论文原文
  • 构建显微图像-文本关联的知识图谱,实现跨模态实体对齐
  • 推理时注入结构化知识,使模型在显微视觉问答中准确率提升37.5%
  • 无需微调即可泛化到多个生物医学任务,适合科研型应用

多模态大语言模型在科学推理中潜力巨大,但在显微领域受限于专业数据稀缺与专家知识难以编码。为此,我们提出MicroWorld框架,从大规模科学图像-标题语料中构建多模态属性图(MAPG),并在推理时无须微调地增强模型能力。该图谱通过scispaCy或基于LLM的三元组挖掘提取约11.1万个节点、34.6万条带类型边,覆盖8类关系,利用Qwen3-VL-Embedding实现图像与实体在共享嵌入空间中的对齐。推理时,通过图检索匹配查询实体,并将结构化知识注入提示。在MicroVQA基准上,使Qwen3-VL-8B-Instruct性能提升37.5%,优于GPT-5达13.0%,达到新最优;在MicroBench上亦提升6.0%。实验验证了其泛化能力,案例分析揭示了知识增强机制及未来改进方向。代码与数据已开源。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) show remarkable potential for scientific reasoning, yet their performance in specialized domains such as microscopy remains limited by the scarcity of domain-specific training data and the difficulty of encoding fine-grained expert knowledge into model parameters. To bridge the gap, we introduce MicroWorld, a framework that constructs a multimodal attributed property graph (MAPG) from large-scale scientific image--caption corpora and leverages it to augment MLLM reasoning at inference time without any domain-specific fine-tuning. MicroWorld extracts biomedical entities and relations via scispaCy or LLM-based triplet mining, aligns images and entities in a shared embedding space using Qwen3-VL-Embedding, and assembles a knowledge graph comprising approximately 111K nodes and 346K typed edges spanning eight relation categories. At inference time, a graph-augmented retrieval pipeline matches query entities to the MAPG and injects structured knowledge context into the MLLM prompt. On the MicroVQA benchmark, MicroWorld improves the reasoning performance of Qwen3-VL-8B-Instruct by 37.5%, outperforming GPT-5 by 13.0% to achieve a new state-of-the-art. Furthermore, it yields a 6.0% performance gain on the MicroBench benchmark. Extensive experiments demonstrate the enhanced generalization capability introduced by MicroWorld. A qualitative case study further reveals both the mechanisms through which structured knowledge improves reasoning and the failure modes that point to promising future directions. Code and data are available at https://github.com/ieellee/MicroWorld.

多模态知识图谱显微图像推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。