用视觉模型自动评代码,三轮迭代提升前端生成质量17.8%。
Vision-Guided Iterative Refinement for Frontend Code Generation

- 引入视觉语言模型作为视觉评判器,反馈网页渲染结果
- 三轮迭代使真实用户请求下性能提升17.8%
- 参数高效微调可保留25%的改进效果,适合资源受限场景
大型语言模型在代码生成中常依赖多阶段人工迭代优化,虽有效但成本高昂,尤其在前端开发这类依赖可视化输出的领域。本文提出一种全自动的“批判者在环”框架,利用视觉-语言模型对生成网页的渲染结果进行结构化反馈,指导代码的迭代优化。在WebDev Arena数据集的真实用户请求上,该方法实现了持续的质量提升,三轮迭代后性能最高提升17.8%。进一步研究发现,通过LoRA进行参数高效微调,可在不显著增加令牌数的前提下,实现最佳批判者方案25%的性能增益。结果表明,基于视觉模型的自动化批判能显著优于单次推理,凸显了迭代优化在复杂视觉输出任务中的重要性。
原文摘要 · Abstract (English)
Code generation with large language models often relies on multi-stage human-in-the-loop refinement, which is effective but very costly - particularly in domains such as frontend web development where the solution quality depends on rendered visual output. We present a fully automated critic-in-the-loop framework in which a vision-language model serves as a visual critic that provides structured feedback on rendered webpages to guide iterative refinement of generated code. Across real-world user requests from the WebDev Arena dataset, this approach yields consistent improvements in solution quality, achieving up to 17.8% increase in performance over three refinement cycles. Next, we investigate parameter-efficient fine-tuning using LoRA to understand whether the improvements provided by the critic can be internalized by the code-generating LLM. Fine-tuning achieves 25% of the gains from the best critic-in-the-loop solution without a significant increase in token counts. Our findings indicate that automated, VLM-based critique of frontend code generation leads to significantly higher quality solutions than can be achieved through a single LLM inference pass, and highlight the importance of iterative refinement for the complex visual outputs associated with web development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。