arXiv:2507.22827cs.CV2025-07被引 25

分阶段拆解界面转代码,提升复杂页面生成准确率。

ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents

  • 三阶段模块化设计:定位、规划、生成由专用智能体分工完成。
  • 在UI生成任务中实现布局准确率与代码正确率的最新突破。
  • 适合前端自动化、低代码工具开发者参考使用。

将用户界面(UI)设计自动转换为前端代码,能显著加速软件开发并推动设计流程民主化。尽管多模态大语言模型(MLLM)可实现图像到代码的转换,但在复杂界面下常因无法统一视觉感知、布局规划与代码生成,导致频繁出错。为此,我们提出ScreenCoder,一个模块化多智能体框架,将任务分解为三个可解释阶段:定位、规划和生成。通过为各阶段分配专用智能体,该框架显著提升了鲁棒性和代码保真度。此外,ScreenCoder作为可扩展的数据引擎,生成高质量图像-代码对,用于微调开源MLLM,采用监督微调与强化学习双阶段流程,大幅增强其界面生成能力。大量实验表明,该方法在布局准确性、结构一致性和代码正确性上均达到当前最优水平。代码已开源:https://github.com/leigest519/ScreenCoder。

原文摘要 · Abstract (English)

Automating the transformation of user interface (UI) designs into front-end code holds significant promise for accelerating software development and democratizing design workflows. While multimodal large language models (MLLMs) can translate images to code, they often fail on complex UIs, struggling to unify visual perception, layout planning, and code synthesis within a single monolithic model, which leads to frequent perception and planning errors. To address this, we propose ScreenCoder, a modular multi-agent framework that decomposes the task into three interpretable stages: grounding, planning, and generation. By assigning these distinct responsibilities to specialized agents, our framework achieves significantly higher robustness and fidelity than end-to-end approaches. Furthermore, ScreenCoder serves as a scalable data engine, enabling us to generate high-quality image-code pairs. We use this data to fine-tune open-source MLLM via a dual-stage pipeline of supervised fine-tuning and reinforcement learning, demonstrating substantial gains in its UI generation capabilities. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in layout accuracy, structural coherence, and code correctness. Our code is made publicly available at https://github.com/leigest519/ScreenCoder.

界面生成多模态智能体系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。