专为设计转代码优化,提升准确率与可复现性。
LOCOFY Large Design Models -- Design to code conversion solution
- 针对设计图构建专用大模型,分步解决布局、元素识别与组件复用问题。
- 在节点定位、响应式布局和可复现性上优于通用大模型。
- 适合前端开发、自动化生成场景,尤其看重代码质量与可维护性。
尽管大型语言模型(LLMs)和多模态大语言模型快速发展,其在设计转代码领域的可解释性、可扩展性、资源消耗及可复现性仍面临挑战。为此,我们提出专为设计与网页训练的大型设计模型(LDMs)范式,实现从设计到代码的无缝转换。通过构建包含数据工程与模型架构改进的训练与推理流程:1)设计优化器利用自有真实数据集修正低效设计;2)基于预训练与微调模型的标签与特征检测,精准识别分类界面元素;3)自动组件提取重复结构为可复用模块,减少冗余并提升代码复用性。推理流程对真实设计生成精确、可解释的代码生成指令,确保可靠性。实验表明,采用新型预览匹配评分指标,LDMs在端到端设计转代码任务中表现优异,相比通用大模型,在节点定位、响应式与可复现性上均具优势。自研标签与特征检测模型在大量测试设计中展现出高精度与一致性。因此,所提出的LDMs是理解设计并生成高效、可靠生产级代码的可靠且优越解决方案。
原文摘要 · Abstract (English)
Despite rapid advances in Large Language Models and Multimodal Large Language Models (LLMs), numerous challenges related to interpretability, scalability, resource requirements and repeatability remain, related to their application in the design-to-code space. To address this, we introduce the Large Design Models (LDMs) paradigm specifically trained on designs and webpages to enable seamless conversion from design-to-code. We have developed a training and inference pipeline by incorporating data engineering and appropriate model architecture modification. The training pipeline consists of the following: 1)Design Optimiser: developed using a proprietary ground truth dataset and addresses sub-optimal designs; 2)Tagging and feature detection: using pre-trained and fine-tuned models, this enables the accurate detection and classification of UI elements; and 3)Auto Components: extracts repeated UI structures into reusable components to enable creation of modular code, thus reducing redundancy while enhancing code reusability. In this manner, each model addresses distinct but key issues for design-to-code conversion. Separately, our inference pipeline processes real-world designs to produce precise and interpretable instructions for code generation and ensures reliability. Additionally, our models illustrated exceptional end-to-end design-to-code conversion accuracy using a novel preview match score metric. Comparative experiments indicated superior performance of LDMs against LLMs on accuracy of node positioning, responsiveness and reproducibility. Moreover, our custom-trained tagging and feature detection model demonstrated high precision and consistency in identifying UI elements across a wide sample of test designs. Thus, our proposed LDMs are a reliable and superior solution to understanding designs that subsequently enable the generation of efficient and reliable production-ready code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。