为电信领域图像转UML代码设计评估指标，量化模型转换准确率。
Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs
- 用版本控制工具对比VLM生成的puml与人工标注真值
- 发现节点、消息等基础结构准确，复杂结构如注释框误差大
- 适合关注AI生成代码质量评估的研究者和工程师
电信领域的3GPP文档包含大量时序图，视觉语言大模型（VLMs）可将其转换为机器可读的PlantUML（puml）格式。但现有方法缺乏对转换结果的细致评估，未系统比较不同组件的表现。本文构建了来自3GPP文档的时序图数据集，用于代表实际场景。对比Claude Sonnet与GPT-4V生成的puml脚本与人工标注的真值，利用版本控制工具捕捉差异，提出标准性能指标，评估参与者识别、消息流准确性、序列顺序及分组结构保留等维度。实验表明，节点、边和消息被准确捕获，但模型在注释、框结构、分组等复杂构造上表现不佳。结果说明，当前训练数据需强化对这些组件的表征，以提升模型泛化能力。
原文摘要 · Abstract (English)
Telecom domain 3GPP documents are replete with images containing sequence diagrams. Advances in Vision-Language Large Models (VLMs) have eased conversion of such images to machine-readable PlantUML (puml) formats. However, there is a gap in evaluation of such conversions - existing works do not compare puml scripts for various components. In this work, we propose performance metrics to measure the effectiveness of such conversions. A dataset of sequence diagrams from 3GPP documents is chosen to be representative of domain-specific actual scenarios. We compare puml outputs from two VLMs - Claude Sonnet and GPT-4V - against manually created ground truth representations. We use version control tools to capture differences and introduce standard performance metrics to measure accuracies along various components: participant identification, message flow accuracy, sequence ordering, and grouping construct preservation. We demonstrate effectiveness of proposed metrics in quantifying conversion errors across various components of puml scripts. The results show that nodes, edges and messages are accurately captured. However, we observe that VLMs do not necessarily perform well on complex structures such as notes, box, groups. Our experiments and performance metrics indicates a need for better representation of these components in training data for fine-tuned VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。