将手机应用小部件图像转为可执行代码，解决视觉紧凑、无上下文难题。
Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
- 按设计原则拆解组件，用图标检索与可视化模块提升识别能力。
- 自研工具链支持多框架输出，生成代码视觉一致性显著提升。
- 专为小部件优化，适合移动开发与自动化界面生成场景。
用户界面到代码（UI2Code）旨在生成能准确重建输入界面的可执行代码。以往研究主要集中于网页和移动端界面，而应用小部件尚未被充分探索。与具有丰富层级结构的网页或移动界面不同，小部件是紧凑、无上下文的微型界面，通过密集布局和图标在严格空间约束下呈现关键信息。此外，尽管网页或移动端的（图像, 代码）配对数据广泛可用，但小部件设计为专有内容，缺乏可访问的标记信息。我们首次将此场景形式化为小部件到代码（Widget2Code），并引入一个仅基于图像的小部件基准测试，包含细粒度、多维度评估指标。基准测试表明，尽管通用多模态大语言模型（MLLMs）优于专用的UI2Code方法，但仍会产生不可靠且视觉不一致的代码。为此，我们提出一个基线系统，同时提升感知理解与结构化代码生成能力。在感知层面，我们遵循小部件设计原则，将原子组件组合成完整布局，配备图标检索与可复用可视化模块。在系统层面，我们设计端到端架构WidgetFactory，包括与框架无关的小部件定制领域特定语言（WidgetDSL）及编译器，可将其转换为多种前端实现（如React、HTML/CSS）。自适应渲染模块进一步优化空间尺寸以满足紧凑性要求。这些贡献显著提升视觉保真度，建立了一个强有力的基线和统一基础设施，为未来小部件代码生成研究奠定基础。
原文摘要 · Abstract (English)
User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and reusable visualization modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML/CSS). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。