arXiv:2411.03292cs.SEcs.AI2024-11被引 42

首个评估大模型生成交互网页的基准,揭示其在动态交互上的四大短板。

Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping

  • 构建交互式网页生成任务与包含374种交互的基准数据集
  • 发现大模型生成交互代码成功率仅62.3%,对细微交互识别差
  • 提出四种增强策略,显著提升交互生成准确率

多模态大语言模型(MLLM)在从静态界面图生成代码方面表现优异,但现有基准仅涵盖静态网页,忽略动态交互,限制了生成网页的实用性与用户参与度。为此,本文首次系统研究MLLM在生成交互式网页方面的能力,提出“交互到代码”任务,并建立Interaction2Code基准,涵盖127个独特网页、374种不同交互,覆盖15类网页类型和31种交互类别。通过使用最先进MLLM进行全面实验,结合自动指标与人工评估,发现四大关键局限:(1)交互代码生成量远低于完整页面,(2)易出现十类失败模式,(3)对视觉细微交互识别能力弱,(4)仅依赖单模态视觉描述时交互理解不足。为此,提出四种改进策略:交互元素高亮、故障感知提示(FAP)、视觉显著性增强、视觉-文本描述融合,均旨在提升MLLM在交互生成任务中的表现。该基准与代码已开源于https://github.com/WebPAI/Interaction2Code。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs' performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.

多模态模型交互生成网页代码基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。