arXiv:2509.24709cs.CV2025-09被引 9

评测大模型从用户操作视频重建网页的能力,发现当前模型逻辑推理严重不足。

IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?

  • 构建视频+静态资源的交互网页重建基准,涵盖100个真实网站的1001个操作
  • 最佳模型功能正确率仅24.39%,远低于视觉保真度64.25%
  • 适合关注多模态推理与代码生成的开发者和研究者

网页转代码任务要求模型理解网页的视觉表示并生成对应代码。然而,现有基准主要关注静态截图转代码,忽视了真实网页应用中动态交互的核心作用。为此,本文提出IWR-Bench,一个评估大视觉语言模型(LVLMs)从用户交互视频重构交互式网页能力的新基准。IWR-Bench包含来自100个真实网站的113项精心设计的任务,共1,001个操作,涵盖多种交互复杂性(如网页游戏)、视觉风格和领域。每个任务均提供用户交互视频及所有爬取的静态资源(如图片、视频),符合标准网页开发实践。该基准评估模型在两个核心挑战上的表现:从视频与资源中进行综合多模态推理以推断交互逻辑,以及将该逻辑转化为可运行代码。采用代理作为裁判的框架与综合指标体系,自动评估生成网页的功能正确性与视觉保真度。对28个LVLM的广泛实验表明,当前模型面临显著挑战:最优模型整体得分仅为36.35%,其中功能正确率(IFS)为24.39%,远低于视觉保真度(VFS)64.25%。这些结果凸显了现有模型在理解时序动态与合成事件驱动逻辑方面的关键缺陷,确立了IWR-Bench作为视觉语言研究的重要前沿。基准与评估代码将公开于https://github.com/SIGMME/IWR-Bench。

原文摘要 · Abstract (English)

The webpage-to-code task requires models to understand visual representations of webpages and generate corresponding code. However, existing benchmarks primarily focus on static screenshot-to-code tasks, thereby overlooking the dynamic interactions fundamental to real-world web applications. To address this limitation, this paper introduces IWR-Bench, a novel benchmark for evaluating the capabilities of Large Vision-Language Models (LVLMs) in interactive webpage reconstruction from video. IWR-Bench comprises 113 meticulously curated tasks from 100 real-world websites, with 1,001 actions and featuring diverse interaction complexities (e.g., web games), visual styles, and domains. Aligning with standard web development practices, each task includes not only user interaction videos but also all crawled static assets (e.g., images, videos). This benchmark evaluates models on two fundamental challenges: comprehensive multi-modal reasoning to infer interaction logic from video and assets, and advanced code generation to translate this logic into functional code. An agent-as-a-judge framework with a comprehensive metric system automatically assesses the functional correctness and visual fidelity of generated webpages. Extensive experiments on 28 LVLMs reveal a significant challenge: the best model achieves an overall score of only 36.35%, as functional correctness (24.39% IFS) lags significantly behind visual fidelity (64.25% VFS). These results highlight critical limitations in current models' ability to reason about temporal dynamics and synthesize event-driven logic, establishing IWR-Bench as a challenging frontier for vision-language research. The benchmark and evaluation code will be made publicly available at https://github.com/SIGMME/IWR-Bench.

网页生成多模态代码生成视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。