用多模态图学习自动合并设计中的碎片化图层，提升代码可读性。
Fragmented Layer Grouping in GUI Designs Through Graph Learning Based on Multimodal Information
- 基于多模态信息构建图神经网络，联合分类与框回归实现分组。
- 在两个真实数据集上达到当前最优性能，显著减少代码碎片。
- 适合前端自动化工具开发者，助力生成更易维护的代码。
自动生成不同粒度的GUI组件分组是实现GUI设计与开发自动化的重要一步。在工业级GUI转代码流程中，碎片化图层会降低生成代码的可读性和可维护性，通过在设计原型中对语义一致的碎片化图层进行分组可缓解此问题。本文提出一种基于图学习的方法，利用设计原型中的多模态信息解决碎片化图层分组问题。模型包含自注意力与图神经网络模块，以融合后的多模态图层表征为输入，创新性地通过联合分类与边界框回归实现图层分组。在两个真实世界数据集上的实验表明，该方法性能达到当前最优。进一步用户研究表明，该方法能有效辅助智能下游工具生成更具可读性和可维护性的前端代码。
原文摘要 · Abstract (English)
Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。