打造可自动生成视觉推理数据的标注工具,让模型更懂图像中的空间与逻辑关系。
From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models

- 用统一原子结构整合位置、语义和拓扑关系,实现多维度信息融合。
- 采用在线策略标注循环,使流程图标注成功率接近100%,每图标注时间持续下降。
- 支持模板化生成多任务推理数据,减少重复标注,适合训练高级视觉语言模型。
视觉语言模型正向复杂的具身结构化视觉推理方向发展。训练此类能力需要一种新型数据,能将空间坐标、开放词汇描述、结构属性和拓扑关系统一为单一表征。然而,现有标注工具存在三大系统性瓶颈:表达力有限、标注与训练脱节、数据复用性差。为此,我们提出开源标注工具 ScreenAnnotator。首先定义统一标注原子模式,将空间、语义和结构原语绑定为单一单元;其次实现嵌入贝叶斯标注验证器(BAV)的在线策略标注循环;最后设计基于模板的多任务数据合成流程,动态将静态原子转化为多样化的多维推理任务,消除重复标注。该在线循环使流程图标注接受率接近100%,GUI截图达77%;随着标注数据积累,单图标注时间持续下降。在流程图场景中,微调后的视觉语言模型平均准确率达76.1%,绝对提升35.1个百分点。代码已公开:https://github.com/WnQinm/Annotator。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual reasoning. Training models for such advanced capabilities demands a new genre of data that seamlessly unifies spatial coordinates, open-vocabulary descriptions, structured attributes, and topological relationships into a singular representation. However, existing data annotation tools fundamentally fail to meet these intricate demands, suffering from three systematic bottlenecks: limited expressiveness, severe annotation-training decoupling, and poor data reusability. To bridge this infrastructure gap, we introduce an open-source annotation tool, ScreenAnnotator. First, we define a unified annotation atom schema that binds spatial, semantic, and structural primitives into a single unit. Second, we implement an on-policy annotation loop embedded with a Bayesian Annotation Verifier (BAV). Finally, we design a template-driven multi-task data synthesis process dynamically transforms static atoms into diverse multi-dimensional reasoning tasks, eliminating redundant re-annotation. The on-policy loop drives the annotation accept rate to nearly 100% on flowcharts and 77% on GUI screenshots, while steadily reducing per-image annotation time as labeled data accumulate. In the flowchart scenario, fine-tuning a VLM yields 76.1% average accuracy, which is a 35.1% point absolute gain. Our code is available at: https://github.com/WnQinm/Annotator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。