用合成数据训练视觉语言模型，提升对技术手绘图的识别与理解能力。
TechING: Towards Real World Technical Image Understanding via VLMs
- 构建真实风格的手绘技术图合成数据集，用于训练视觉语言模型。
- 在真实手绘图上实现6.97倍的F1分数提升，平均编译错误最少。
- 适合需要处理技术图纸的工程师、研究人员及自动化工具开发者。
技术人员在讨论中常手绘技术图（如流程图、框图等），但后续编辑需重新绘制。现有视觉语言模型（VLMs）难以理解此类图像，而真实手绘数据难以大规模生成。本文提出一个大规模合成数据集，模拟真实场景中的手绘技术图，并设计多种自监督任务进行训练。基于此，我们微调 Llama 3.2 11B-instruct 模型，得到 LLama-VL-TUG，其在基准测试中将原始模型的 ROUGE-L 分数提升 2.14 倍，综合表现最优。在真实手绘图像上的人工评估显示，该模型在 8 种图类型中的 7 种达到最低编译错误率，平均 F1 分数相较原模型提升 6.97 倍。
原文摘要 · Abstract (English)
Professionals working in technical domain typically hand-draw (on whiteboard, paper, etc.) technical diagrams (e.g., flowcharts, block diagrams, etc.) during discussions; however, if they want to edit these later, it needs to be drawn from scratch. Modern day VLMs have made tremendous progress in image understanding but they struggle when it comes to understanding technical diagrams. One way to overcome this problem is to fine-tune on real world hand-drawn images, but it is not practically possible to generate large number of such images. In this paper, we introduce a large synthetically generated corpus (reflective of real world images) for training VLMs and subsequently evaluate VLMs on a smaller corpus of hand-drawn images (with the help of humans). We introduce several new self-supervision tasks for training and perform extensive experiments with various baseline models and fine-tune Llama 3.2 11B-instruct model on synthetic images on these tasks to obtain LLama-VL-TUG, which significantly improves the ROUGE-L performance of Llama 3.2 11B-instruct by 2.14x and achieves the best all-round performance across all baseline models. On real-world images, human evaluation reveals that we achieve minimum compilation errors across all baselines in 7 out of 8 diagram types and improve the average F1 score of Llama 3.2 11B-instruct by 6.97x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。