让AI通过对比截图与生成代码的视觉差异来优化代码,提升还原度。
VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation
- 基于视觉差异设计监督信号,关联外观变化与代码修改
- 引入强化学习实现自我修正,显著提升布局还原度
- 适合需要高保真界面还原的开发自动化场景
截图转代码旨在将用户界面截图转化为可执行的前端代码,以准确复现目标布局与样式。现有多模态大模型直接从截图映射代码,但训练过程中未观察生成代码的实际视觉效果。相比之下,人类开发者会反复渲染代码、对比设计图,并学习视觉差异与代码变更的关系。受此启发,我们提出VisRefiner,一种让模型从渲染结果与参考设计间的视觉差异中学习的训练框架。我们构建了差异对齐的监督信号,将视觉偏差与对应的代码修改关联起来,使模型理解外观变化如何源于实现差异。在此基础上,引入强化学习阶段进行自我优化:模型通过观察渲染输出与目标设计的视觉差异,动态调整代码。实验表明,VisRefiner显著提升了单步生成质量与布局保真度,并赋予模型强大的自我修正能力。结果验证了从视觉差异中学习对推进截图转代码的有效性。
原文摘要 · Abstract (English)
Screenshot-to-code generation aims to translate user interface screenshots into executable frontend code that faithfully reproduces the target layout and style. Existing multimodal large language models perform this mapping directly from screenshots but are trained without observing the visual outcomes of their generated code. In contrast, human developers iteratively render their implementation, compare it with the design, and learn how visual differences relate to code changes. Inspired by this process, we propose VisRefiner, a training framework that enables models to learn from visual differences between rendered predictions and reference designs. We construct difference-aligned supervision that associates visual discrepancies with corresponding code edits, allowing the model to understand how appearance variations arise from implementation changes. Building on this, we introduce a reinforcement learning stage for self-refinement, where the model improves its generated code by observing both the rendered output and the target design, identifying their visual differences, and updating the code accordingly. Experiments show that VisRefiner substantially improves single-step generation quality and layout fidelity, while also endowing models with strong self-refinement ability. These results demonstrate the effectiveness of learning from visual differences for advancing screenshot-to-code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。