测试视觉模型能否从视频中还原动态代码，发现现有模型难保时间一致性。
Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation
- 构建视频转动态代码的基准数据集，含1069段动画视频与对应代码。
- 提出外观相似度与时间相似度双指标，分离视觉质量与时间对齐评估。
- 即使外观还原好，主流模型仍严重丢失动作时序，适合研究视频理解者看。
尽管近期视觉语言模型（VLMs）在静态视觉转代码任务（如网页、图表或SVG生成）上取得显著进展，但其在存在运动时是否能恢复时序动态仍不明确。为此，我们提出Animation2Code，一个通过从视频重建可执行网页动画代码来评估时序视觉推理的基准。该数据集包含1,069个具有多样化视觉外观和运动模式的网页动画视频，均配有对应的HTML/CSS/JavaScript实现。我们提出两个与人类判断对齐的指标：外观相似度与时间相似度，可将视觉保真度与时间对齐性分开评估。在该数据集上对当前最先进的VLMs进行基准测试发现，即便在高外观相似度下，模型仍难以维持重建的时间一致性，包括微调和迭代优化设置中也表现不佳。代码与数据可在https://anya-ji.github.io/animation2code-website 获取。
原文摘要 · Abstract (English)
While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it remains unclear whether they can recover temporal dynamics when motion is present. To this end, we introduce Animation2Code, a benchmark for evaluating temporal visual reasoning via reconstructing executable web animation code from videos. Animation2Code consists of 1,069 web animation videos with diverse visual appearances and motion patterns, paired with corresponding HTML/CSS/JavaScript implementations. We propose two human-aligned metrics, appearance similarity and temporal similarity, which allow us to disentangle visual fidelity from temporal alignment when comparing rendered animations against ground-truth samples. Benchmarking state-of-the-art VLMs on this dataset shows that current VLMs struggle to maintain temporal consistency in reconstruction, even when achieving high appearance similarity, including under finetuning and iterative refinement settings. Code and data are available at https://anya-ji.github.io/animation2code-website .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。