arXiv:2503.21889cs.CVcs.AI2025-03Conference of the …被引 3

用草图自动生成可执行工作流,提升企业自动化效率。

StarFlow: Generating Structured Workflow Outputs From Sketch Images

  • 基于视觉语言模型,将手绘草图转为结构化工作流。
  • 微调后模型性能显著优于通用大模型,准确率明显提升。
  • 适合需要快速构建自动化流程的开发者与企业用户。

工作流是企业平台自动化的核心,用于任务编排、数据处理和系统集成。尽管广泛应用,构建工作流仍复杂,通常需通过低代码平台或可视化编程工具手动配置。为简化此过程,我们探索利用生成式基础模型,特别是视觉语言模型(VLMs),从视觉输入自动生成结构化工作流。将手绘草图或计算机生成的图表转换为可执行工作流面临挑战:自由形式绘画的模糊性、图表风格差异,以及从视觉元素推断执行逻辑的困难。为此,我们提出StarFlow框架,利用视觉语言模型从草图生成结构化工作流。我们构建了一个包含合成、人工标注和真实样本的多样化工作流图数据集,以支持稳健训练与评估。对多个视觉语言模型进行微调并开展消融实验,分析方法优劣。结果表明,微调显著提升了结构化工作流生成能力,在该任务上超越了大型视觉语言模型。

原文摘要 · Abstract (English)

Workflows are a fundamental component of automation in enterprise platforms, enabling the orchestration of tasks, data processing, and system integrations. Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. To simplify this process, we explore the use of generative foundation models, particularly vision-language models (VLMs), to automatically generate structured workflows from visual inputs. Translating hand-drawn sketches or computer-generated diagrams into executable workflows is challenging due to the ambiguity of free-form drawings, variations in diagram styles, and the difficulty of inferring execution logic from visual elements. To address this, we introduce StarFlow, a framework for generating structured workflow outputs from sketches using vision-language models. We curate a diverse dataset of workflow diagrams -- including synthetic, manually annotated, and real-world samples -- to enable robust training and evaluation. We finetune and benchmark multiple vision-language models, conducting a series of ablation studies to analyze the strengths and limitations of our approach. Our results show that finetuning significantly enhances structured workflow generation, outperforming large vision-language models on this task.

工作流生成视觉语言模型自动化草图理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。