构建多模态网页编码评估基准,全面测试模型生成、编辑与修复能力。
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models

- 基于文本、图像、视频输入,覆盖生成、编辑、修复三类任务七种场景。
- 采用真实浏览器执行与自动化测试,评估网页交互性与视觉质量表现。
- 揭示开源模型在美观性与交互维持上仍存短板,框架选择显著影响结果。
大语言模型正快速演变为具备端到端网页编码能力的交互式编程代理,但现有评测基准仅聚焦于静态文本生成与语法正确性,忽视了视觉保真度、交互质量及代码库级推理。为此,我们提出 WebCompass,一个统一的多模态网页工程能力评估基准。考虑到真实网页开发是生成、编辑与修复的迭代过程,该基准涵盖三种输入模态(文本、图像、视频)和三种任务类型(生成、编辑、修复),形成七类任务,贴近实际工作流。通过多阶段人工参与的流水线,我们构建了覆盖15个生成领域、16种编辑操作类型和11种修复缺陷类型的实例,每项标注为易/中/难三个难度级别。评估方面,采用检查清单引导的LLM作为裁判协议评估编辑与修复任务,并提出新颖的Agent-as-a-Judge范式用于生成任务:自主在真实浏览器中执行生成网站,利用模型上下文协议(MCP)探索交互行为,并迭代生成针对性测试用例,高度逼近人工验收测试。我们评估了代表性闭源与开源模型,发现:(1) 闭源模型整体更强且更均衡;(2) 编辑与修复表现出不同难度特征,修复虽更擅长保持交互性但仍面临执行挑战;(3) 美学是持续瓶颈,尤其对开源模型影响显著;(4) 框架选择显著影响结果,Vue始终更具挑战性,而React与Vanilla/HTML在不同任务中表现更优。
原文摘要 · Abstract (English)
Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability, typically text-conditioned generation with static-correctness metrics, leaving visual fidelity, interaction quality, and codebase-level reasoning largely unmeasured. We introduce WebCompass, a multimodal benchmark that provides unified lifecycle evaluation of web engineering capability. Recognizing that real-world web coding is an iterative cycle of generation, editing, and repair, WebCompass spans three input modalities (text, image, video) and three task types (generation, editing, repair), yielding seven task categories that mirror professional workflows. Through a multi-stage, human-in-the-loop pipeline, we curate instances covering 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy/Medium/Hard levels. For evaluation, we adopt a checklist-guided LLM-as-a-Judge protocol for editing and repair, and propose a novel Agent-as-a-Judge paradigm for generation that autonomously executes generated websites in a real browser, explores interactive behaviors via the Model Context Protocol (MCP), and iteratively synthesizes targeted test cases, closely approximating human acceptance testing. We evaluate representative closed-source and open-source models and observe that: (1) closed-source models remain substantially stronger and more balanced; (2) editing and repair exhibit distinct difficulty profiles, with repair preserving interactivity better but remaining execution-challenging; (3) aesthetics is the most persistent bottleneck, especially for open-source models; and (4) framework choice materially affects outcomes, with Vue consistently challenging while React and Vanilla/HTML perform more strongly depending on task type.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。