首个评估视觉语言模型理解实体关系图的基准,揭示现有模型在复杂结构上表现差。
ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams

- 构建包含2960张图表的大规模基准,覆盖多种领域与复杂度
- 模型对弱实体等复杂结构识别率低至0.07 F1,整体性能受限于语义先验
- 适合数据库智能设计、多模态理解研究者使用
实体-关系图(ERD)是概念数据库设计的核心,但通常仅以渲染图像形式存在,缺乏机器可读性,限制了AI辅助数据库工程的发展。我们提出ERUnderstand,首个面向结构化ER图理解的大规模基准,包含从教育资料、真实世界模式及合成生成中收集的2,960张图表,涵盖多样领域、记法、复杂度及扩展实体-关系(EER)结构。每张图均配以标准化机器可读表示,支持细粒度的模式元素评估。对前沿视觉语言模型(VLMs)的评估发现,常见ERD元素可被可靠恢复(F1 > 0.74),但在弱实体(最低0.28 F1)、多值属性(0.14 F1)和N元关系(0.07 F1)上性能显著下降。引入推理增强的模型可提升15%-25%整体表现,但仍受语言先验和图复杂度影响。ERUnderstand为概念数据库模式的多模态理解提供了标准化评估框架。该基准、数据集、评估工具包与生成代码已开源:https://github.com/salinaria/ERUnderstand。
原文摘要 · Abstract (English)
Entity-Relationship Diagrams (ERDs) are central to conceptual database design, yet they are typically available only as rendered images rather than machine-readable schemas, limiting AI-assisted database engineering. We introduce ERUnderstand, the first large-scale benchmark for structured understanding of ER diagrams, comprising 2,960 diagrams collected from curated educational sources, real-world schemas, and synthetically generated examples spanning diverse domains, notations, complexity levels, and Extended Entity-Relationship (EER) constructs. Each diagram is paired with a standardized machine-readable representation for fine-grained evaluation of schema elements. Evaluating state-of-the-art Vision-Language Models (VLMs), we find that while common ERD elements are recovered reliably (F1 > 0.74), performance drops sharply on weak entities (as low as 0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). Reasoning-augmented models improve overall performance by 15-25% but remain sensitive to linguistic priors and increasing diagram complexity. ERUnderstand provides a standardized benchmark for evaluating multimodal understanding of conceptual database schemas. The benchmark, dataset, evaluation toolkit, and generation code are publicly available at https://github.com/salinaria/ERUnderstand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。