测试发现大模型解读类图时更依赖预训练先验而非真实图示。
Prior Bias in Vision Language Models on UML Diagram Interpretation
- 设计对比实验：相同布局下仅反转关系箭头，检验模型是否真看图
- 开源模型平均准确率下降33.48%，三类场景下降幅达45.28%
- 闭源模型仍有10%差距，规模扩大也难改先验依赖问题
视觉语言模型（VLMs）被广泛用于软件工程中的可视化产物，尤其是依赖视觉符号的UML类图。然而，这些模型是真正理解图示，还是仅凭预训练中对类关系的先验认知作答仍不明确。为此，我们构建了一个可控的UML基准测试：每个符合先验的图都配有一个冲突图，二者保持相同的类名和布局，仅反转关系箭头。我们评估了来自InternVL3.5和Qwen3两个模型家族的八款开源VLM，以及两款闭源前沿模型GPT-5.4和GPT-5.4 Mini。在八款开源模型中，箭头反转导致关系方向准确率平均下降33.48%；在更复杂的三类情形下，开源模型准确率骤降45.28%，而GPT-5.4家族平均下降18.62%。模型规模扩展带来的提升有限且因家族而异。该基准揭示了模型在图示驱动的软件理解中存在诊断性先验偏差。相关代码与数据已公开于https://anonymous.4open.science/r/UMLKnowledgeConflict-8461。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) are increasingly applied to software engineering artifacts, especially UML class diagrams whose meaning depends on visual notation. Yet, it is unclear whether VLMs actually read such diagrams or instead answer from pretrained priors about how classes typically relate. We introduce a controlled UML benchmark in which each prior-conforming diagram is paired with its prior-conflicting counterpart that (1) preserves the same class names and layout while (2) reverses only the relation arrow. We evaluate eight open-source VLMs from two model families, InternVL3.5 and Qwen3, alongside two closed-source frontier models GPT-5.4 and GPT-5.4 Mini. Across the eight open-source models, reversing the arrow reduces relation-direction accuracy by 33.48% on average, while GPT-5.4 Mini retains a 10% gap. In the harder three-class condition, accuracy drops sharply by 45.28% for open-source models, and even 18.62% for the GPT-5.4 family on average. Scaling provides only limited improvements and is family-dependent. Our benchmark presents a diagnostic prior-driven failure in diagram-grounded software understanding. Our artifact is available at https://anonymous.4open.science/r/UMLKnowledgeConflict-8461.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。