让AI自动生成可交互的网页代码并验证其正确性
WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
- 用智能体抓取多状态界面截图，驱动代码生成
- 生成的代码能真正运行且支持用户交互，准确率显著提升
- 适合需要自动化开发网页的工程师和研究者
用户界面开发需将设计稿转化为可运行代码，过程重复且耗时。现有视觉语言模型虽能生成静态HTML/CSS/JavaScript页面，但缺乏交互能力。为此，我们提出WebVIA，首个用于交互式UI-to-Code生成与验证的基于网络的智能体框架。该框架包含三个组件：1）探索智能体，用于捕获多状态界面截图；2）UI2Code模型，生成可执行的交互式代码；3）验证模块，确保交互逻辑正确。实验表明，WebVIA-Agent在界面探索中比通用智能体（如Gemini-2.5-Pro）更稳定准确。微调后的WebVIA-UI2Code模型在交互式和静态UI-to-Code基准测试中均显著优于基线模型。代码与模型已开源：https://webvia.github.io。
原文摘要 · Abstract (English)
User interface (UI) development requires translating design mockups into functional code, a process that remains repetitive and labor-intensive. While recent Vision-Language Models (VLMs) automate UI-to-Code generation, they generate only static HTML/CSS/JavaScript layouts lacking interactivity. To address this, we propose WebVIA, the first agentic framework for interactive UI-to-Code generation and validation. The framework comprises three components: 1) an exploration agent to capture multi-state UI screenshots; 2) a UI2Code model that generates executable interactive code; 3) a validation module that verifies the interactivity. Experiments demonstrate that WebVIA-Agent achieves more stable and accurate UI exploration than general-purpose agents (e.g., Gemini-2.5-Pro). In addition, our fine-tuned WebVIA-UI2Code models exhibit substantial improvements in generating executable and interactive HTML/CSS/JavaScript code, outperforming their base counterparts across both interactive and static UI2Code benchmarks. Our code and models are available at \href{https://zheny2751-dotcom.github.io/webvia.github.io/}{\texttt{https://webvia.github.io}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。