让代码生成像设计一样迭代优化，通过视觉反馈持续改进界面代码。
UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization
- 将界面生成代码改为闭环迭代优化，结合运行、查看、修正流程。
- 在多个基准测试中超越更大模型，性能随迭代次数提升。
- 适合需要反复调试界面的开发者，尤其关注真实开发流程。
UI-to-code 旨在将界面截图转换为可执行的前端代码。尽管视觉语言模型（VLMs）已取得进展，但现有方法大多将任务视为单次生成，与真实开发中迭代、反馈驱动的特性不符。本文将 UI-to-code 重新定义为交互式视觉优化问题，将代码生成嵌入执行、视觉检查和基于渲染反馈的迭代优化闭环中。针对视觉目标不可微及绝对评估器噪声问题，提出相对视觉策略优化（RVPO），一种基于偏好强化学习的方法，在执行反馈下优化候选结果的相对视觉排序。我们构建了开源的 9B 模型 UI2Code^N，通过持续预训练、监督微调和强化学习训练。实验表明，该模型在界面草图绘制、润色和编辑任务上达到领先水平，即使优于更大模型，且性能随迭代优化持续提升。代码与模型已公开于 https://github.com/zai-org/UI2Code_N。
原文摘要 · Abstract (English)
UI-to-code aims to translate UI screenshots into executable front-end code. Despite progress with vision-language models (VLMs), most existing methods formulate UI-to-code as a single-pass generation, which mismatches real-world UI development that is inherently iterative and feedback-driven. We reformulate UI-to-code as an interactive visual optimization problem, where code generation is embedded in a closed-loop process of execution, visual inspection, and iterative refinement driven by rendered visual feedback. To address the non-differentiability of visual objectives and the noise of absolute visual evaluators, we propose Relative Visual Policy Optimization (RVPO), a preference-based reinforcement learning method that optimizes relative visual rankings among rendered candidates under execution feedback. We instantiate this paradigm in UI2Code^N, an open-source 9B model trained via continual pre-training, supervised fine-tuning, and reinforcement learning. Experiments demonstrate state-of-the-art performance on UI drafting, UI polishing, and UI editing benchmarks, even outperforming larger models, with performance consistently improving through iterative visual optimization. Our code and models are available at https://github.com/zai-org/UI2Code_N.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。