arXiv:2607.06306cs.SEcs.AI2026-07

首个评估网页交互推理能力的基准,发现模型视觉还原强但交互理解弱。

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

论文配图:UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
图 1 · 摘自论文原文
  • 用截图直接生成可运行网页,不依赖文字提示
  • 6个前沿模型中视觉最强的仅7.5分(满分12.7),交互得分差5.2倍
  • 跨页面状态等复杂交互仍无法实现,一半模型得0分

大语言模型在网页生成方面表现日益出色。然而,现有文本驱动方法依赖复杂提示,对用户要求高且布局表达力有限。图像驱动范式以界面截图为输入,更贴近真实开发流程。但当前基准主要关注视觉保真度,缺乏对生成物交互能力的系统评估。为此,我们提出UI2App,首个聚焦交互推理的基准,即仅凭截图恢复应用行为的能力,无需任何文本或行为引导。UI2App包含327张截图,分为45组状态一致的多路径可运行网页应用。设计端到端评估流程,从可执行性、导航可达性、视觉保真度和交互推理四个维度评测。交互指标(IIS)通过功能正确性和状态管理复杂度评估,奖励任意有效实现而非唯一参考。在六个前沿视觉-语言模型上实验显示,视觉重建与交互实现间存在显著能力差距:视觉保真度领先的模型IIS仅7.5分,排名第四,落后于最高分达5.2倍。高复杂度交互如跨页面状态仍是普遍瓶颈,半数模型在该维度得分为零。总体表明,仅从静态截图推断完整交互行为仍是模型的核心挑战。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.

交互推理网页生成视觉语言模型评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。