arXiv:2503.01619cs.CVcs.AI2025-03被引 15

用合成数据提升视觉语言模型生成前端代码的准确性。

Advancing vision-language models in front-end development via data synthesis

  • 构建自包含代码提取与渲染流水线,生成图文对数据
  • 在React代码生成上达到0.78的pass@k指标,优于基线
  • 适合需要自动化前端开发的工程师和研究者

现代前端开发(如使用React、Vue等框架)面临模块化架构管理、数据与视图同步、组件复用等挑战,导致当前先进视觉语言模型(VLMs)难以直接从设计图生成准确且可运行的代码。为此,我们提出一种反思式智能体工作流,通过合成高质量图像-文本数据来捕捉前端开发的多样性特征。该流程自动从真实项目中提取自包含代码片段,生成对应视觉输出,并生成关联设计元素与功能代码的详细描述。为扩展数据规模与实用性,我们引入三种合成策略:基于演化的合成(实现可扩展、多样化的数据增长)、基于瀑布模型的合成(从系统需求生成逻辑连贯的代码)、以及增量式开发合成(逐步提升人工编写的组件复杂度)。我们基于合成数据训练了大型视觉语言模型Flame,通过pass@k指标验证其在生成React代码上的有效性。结果表明,在代码生成前先理解图像的模型表现更优。

原文摘要 · Abstract (English)

Modern front-end (FE) development, especially when leveraging the unique features of frameworks like React and Vue, presents distinctive challenges. These include managing modular architectures, ensuring synchronization between data and visual outputs for declarative rendering, and adapting reusable components to various scenarios. Such complexities make it particularly difficult for state-of-the-art large vision-language models (VLMs) to generate accurate and functional code directly from design images. To address these challenges, we propose a reflective agentic workflow that synthesizes high-quality image-text data to capture the diverse characteristics of FE development. This workflow automates the extraction of self-contained\footnote{A \textbf{self-contained} code snippet is one that encapsulates all necessary logic, styling, and dependencies, ensuring it functions independently without requiring external imports or context.} code snippets from real-world projects, renders the corresponding visual outputs, and generates detailed descriptions that link design elements to functional code. To further expand the scope and utility of the synthesis, we introduce three data synthesis strategies: Evolution-based synthesis, which enables scalable and diverse dataset expansion; Waterfall-Model-based synthesis, which generates logically coherent code derived from system requirements; and Additive Development synthesis, which iteratively increases the complexity of human-authored components. We build a large vision-language model, Flame, trained on the synthesized datasets and demonstrate its effectiveness in generating React code via the $\text{pass}@k$ metric. Our results suggest that a code VLM trained to interpret images before code generation may achieve better performance.

视觉语言模型前端生成数据合成React

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。