arXiv:2409.18980cs.CLcs.AI2024-09ACL被引 21

构建首个评估图像转网页能力的基准,关注元素完整性和布局准确性。

IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web

  • 提出元素准确率和布局准确率,基于DOM树解析与公共子序列分析。
  • 在1200组图像-网页对上测试,发现现有模型在布局还原上表现不足。
  • 设计五步链式思维提示,提升模型生成网页代码的逻辑性与完整性。

近期大型多模态模型在图像理解方面取得了显著进展,但缺乏针对图像转网页任务的可靠评估基准。现有方法(如BLEU)易受网页中不可见元素干扰,且忽视了页面元素间的空间位置关系。为此,我们构建了图像与对应网页代码的对齐基准集IW-Bench,包含1200对不同难度的数据。提出两种评估指标:通过解析文档对象模型(DOM)树计算元素准确率以衡量元素完整性;将DOM树转换为公共子序列,分析布局准确率以评估元素间位置关系。同时设计五跳多模态链式思维提示(Chain-of-Thought Prompting),包含:1)SoM提示注入,2)元素推断,3)布局推断,4)网页代码生成,5)反思优化。在多个主流大模型上的实验揭示了其在图像转网页任务中的性能瓶颈与改进方向。

原文摘要 · Abstract (English)

Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.

图像转网页多模态评估网页生成链式推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。