用草图生成网页原型，测试大模型理解能力与交互设计效果。
Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping
- 用草图输入测试视觉语言模型的自动网页生成能力。
- 顶级模型仍难准确理解草图，且提问能力有限。
- 设计师更倾向主动提问的交互方式，推动对话式设计发展。
草图是用户界面设计师构思早期想法的自然且易用的方式。然而，现有的用户界面/用户体验自动化研究通常需要高保真输入，如Figma设计或详细截图，限制了可用性并阻碍高效设计迭代。为弥合这一差距，我们提出了Sketch2Code，一个评估前沿视觉语言模型（VLMs）将简单草图转换为网页原型能力的基准。除了端到端评估，Sketch2Code还支持模拟真实设计工作流的交互式代理评估，其中基于VLM的代理通过与模拟用户通信来逐步优化生成结果，可被动接收反馈指令或主动提出澄清问题。我们全面分析了十款商业和开源模型，发现Sketch2Code对现有VLMs极具挑战性；即使最强大的模型也难以准确解读草图，且在生成有效问题以实现持续改进方面表现不佳。然而，与用户界面/用户体验专家进行的用户研究揭示，参与者显著偏好主动提问而非被动接收反馈，凸显了开发更高效多轮对话代理范式的迫切需求。
原文摘要 · Abstract (English)
Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。