用结构感知与对比学习提升模型生成网页代码能力
WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development
- 引入结构感知注意力机制，让模型更好理解HTML层级结构
- 通过对比学习对齐界面图与代码语义，提升生成准确率
- 适合想自动化前端开发的工程师和研究者
网页开发需将UI设计转化为功能页面，对新手和资深开发者均具挑战性，主要因HTML的层次结构复杂及视觉设计与文本代码之间的鸿沟。针对此，我们提出Waffle微调策略：采用结构感知注意力机制增强模型对HTML层级的理解，并使用对比微调方法对齐模型对界面图像与HTML代码的认知。在新构建的WebSight-Test和现有Design2Code基准上，经Waffle微调的模型在HTML匹配度上最高提升9.00个百分点，CW-SSIM提升0.0982，CLIP得分提高32.99，LLEM提升27.12个百分点，显著优于当前主流微调方法。
原文摘要 · Abstract (English)
Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。