arXiv:2507.04952cs.CLcs.SE2025-07被引 29

用自动化多模态评估,精准衡量代码生成的视觉与交互质量。

ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

  • 通过渲染代码生成物并截取动态画面,实现可视化评估
  • 94.4%匹配人工偏好排名,超90%与专家判断一致
  • 适合关注交互式前端生成、模型真实用户体验的研究者

大型语言模型(LLMs)的生成能力正从静态代码扩展到动态交互式视觉产物。然而,现有评估基准聚焦算法正确性,忽视了现代用户体验中的视觉保真度与交互完整性。为此,我们提出ArtifactsBench,一个用于自动化多模态评估视觉代码生成的新基准与范式。该框架通过程序化渲染生成产物,并利用时间序列截图捕捉其动态行为。结合源代码,这些视觉证据由多模态大模型(MLLM)作为评判者进行评估,严格遵循细粒度任务清单,确保全面且可复现的打分。我们构建了一个包含1,825个多样化任务的新基准,评估了30余款主流LLM。自动化评估在排名一致性上达到94.4%,与WebDev Arena(网页开发领域人工偏好金标准)高度一致;与人类专家的成对一致性超过90%。这标志着ArtifactsBench是首个能规模化可靠评估人类感知质量的框架。分析揭示,通用模型常优于领域专用模型。我们已开源ArtifactsBench,包括基准数据、评估工具链及基线结果,网址为https://artifactsbenchmark.github.io/,旨在为社区提供可扩展、高精度的用户中心生成模型研发工具。

原文摘要 · Abstract (English)

The generative capabilities of Large Language Models (LLMs) are rapidly expanding from static code to dynamic, interactive visual artifacts. This progress is bottlenecked by a critical evaluation gap: established benchmarks focus on algorithmic correctness and are blind to the visual fidelity and interactive integrity that define modern user experiences. To bridge this gap, we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation. Our framework programmatically renders each generated artifact and captures its dynamic behavior through temporal screenshots. This visual evidence, alongside the source code, is then assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a fine-grained, per-task checklist to ensure holistic and reproducible scoring. We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading LLMs. Our automated evaluation achieves a striking 94.4% ranking consistency with WebDev Arena, the gold-standard for human preference in web development, and over 90% pairwise agreement with human experts. This establishes ArtifactsBench as the first framework to reliably automate the assessment of human-perceived quality at scale. Our analysis provides a high-resolution map of the current SOTA, revealing that generalist models often outperform domain-specific ones. We open-source ArtifactsBench, including the benchmark, evaluation harness, and baseline results at https://artifactsbenchmark.github.io/, to provide the community with a scalable and accurate tool to accelerate the development of user-centric generative models.

代码生成多模态评估交互式界面自动化测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。