统一理解与生成的UI大模型，提升准确率与生成质量。
UI-UG: A Unified MLLM for UI Understanding and Generation
- 融合SFT与GRPO，增强对复杂UI的细粒度理解。
- 采用DPO优化生成偏好，实现高质量界面输出。
- 工业级流程设计，计算成本低，适合实际应用。
尽管多模态大语言模型已广泛应用于多个领域，但在特定任务如用户界面（UI）理解准确率和生成质量方面仍面临挑战。本文提出UI-UG（一个统一的用于UI理解与生成的多模态大模型），集成双重能力。在理解任务中，采用监督微调（SFT）结合组相对策略优化（GRPO），以提升对现代复杂UI数据的细粒度理解；在生成任务中，进一步使用直接偏好优化（DPO）使模型生成符合人类偏好的界面。此外，我们提出一套工业高效的流程，包括设计适合大模型的领域专用语言（DSL）、训练策略、渲染过程与评估指标。实验表明，该模型在理解任务上达到当前最优（SOTA）性能，优于更大规模的通用多模态大模型及同规模的专业化模型。在界面生成性能上，与更大模型相当，但计算成本仅为几分之一。同时验证了理解与生成任务的联合优化可同步提升两者准确率与质量。代码与模型：https://github.com/neovateai/UI-UG
原文摘要 · Abstract (English)
Although Multimodal Large Language Models (MLLMs) have been widely applied across domains, they are still facing challenges in domain-specific tasks, such as User Interface (UI) understanding accuracy and UI generation quality. In this paper, we introduce UI-UG (a unified MLLM for UI Understanding and Generation), integrating both capabilities. For understanding tasks, we employ Supervised Fine-tuning (SFT) combined with Group Relative Policy Optimization (GRPO) to enhance fine-grained understanding on the modern complex UI data. For generation tasks, we further use Direct Preference Optimization (DPO) to make our model generate human-preferred UIs. In addition, we propose an industrially effective workflow, including the design of an LLM-friendly domain-specific language (DSL), training strategies, rendering processes, and evaluation metrics. In experiments, our model achieves state-of-the-art (SOTA) performance on understanding tasks, outperforming both larger general-purpose MLLMs and similarly-sized UI-specialized models. Our model is also on par with these larger MLLMs in UI generation performance at a fraction of the computational cost. We also demonstrate that integrating understanding and generation tasks can improve accuracy and quality for both tasks. Code and Model: https://github.com/neovateai/UI-UG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。