arXiv:2606.20711cs.CVcs.AI2026-06被引 1

用视频生成网页代码,能精准捕捉操作间的状态变化。

Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit

论文配图:Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit
图 1 · 摘自论文原文
  • 先定位关键操作区域,再高精度重看这些片段
  • 在密集多步操作上功能正确率显著提升
  • 适合需要从视频自动转网页的开发者

UI 视频为生成交互式网页提供了自然输入,因其同时记录了页面外观和操作触发的状态变化。然而,直接应用视频视觉语言模型仍不充分。现有模型通常依赖稀疏采样或压缩时间表示,可能遗漏短时操作边界,破坏实现网页行为所需的状态-动作-状态转换。本文将 UI 视频到代码的生成任务定义为从交互视频中恢复可执行的状态转移,并识别出该失败模式为状态转移错位。提出 Video2Code,一种面向动作感知的视频到代码方法,用于恢复可执行的用户界面状态转移。不同于对视频均匀分配视觉预算,Video2Code 首先进行粗粒度视频理解以定位动作关键区域,随后使用时间裁剪工具在更高时间分辨率下重新审视这些区域,再生成 HTML/CSS/JavaScript 代码。通过动作对齐的视频-代码监督进行实例化,并在视觉与功能标准下评估。实验表明,Video2Code 显著增强了底层开源模型在 UI 视频到代码生成上的能力,在密集多步交互场景中尤其提升了功能正确性。

原文摘要 · Abstract (English)

UI videos provide a natural input for generating interactive webpages, as they capture both webpage appearance and action-triggered state transitions. However, directly applying video-capable vision-language models to this task remains insufficient. Existing models typically rely on sparse sampling or compressed temporal representations, which may miss short action boundaries and break the state-action-state transitions needed to implement webpage behavior. We formulate UI video-to-code generation as executable state-transition recovery from interaction videos, and identify this failure mode as state-transition misalignment. We introduce Video2Code, an action-aware video-to-code approach for recovering executable UI state transitions. Rather than allocating the visual budget uniformly across the video, Video2Code first performs coarse video understanding to locate action-critical regions, then invokes a temporal clipping tool to revisit these regions at higher temporal resolution before generating HTML/CSS/JavaScript code. We instantiate Video2Code with action-aligned video-code supervision and evaluate it under both visual and functional criteria. Experiments show that Video2Code substantially strengthens the underlying open-source model for UI video-to-code generation, improving functional correctness over direct video observation, especially on dense multi-step interactions.

视频生成网页自动化状态迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。