arXiv:2609.02088cs.CV2026-09

让网页生成器通过真实运行反馈自动修复交互错误。

Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development

论文配图:Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
图 1 · 摘自论文原文
  • 用浏览器实时运行代码,根据用户操作反馈迭代修改网页。
  • 在IWR-Bench上将小模型准确率从40.4%提升至57.5%,超越大模型。
  • 适合需要高交互性、可落地的自动化网页开发场景。

多模态大模型在前端开发中已能从截图或交互视频生成互动网页,但现有方法多关注视觉美观与布局相似度,忽视了交互功能的正确性验证。我们提出RILA,一种将浏览器渲染过程纳入闭环的执行驱动型代理,通过运行时交互反馈持续优化生成代码。RILA引入动作交互验证(AIV)模块,在生成网页上重放参考操作轨迹,收集基于执行的实证观察;并设计执行感知渲染评分(ERS),联合衡量交互正确性与视觉保真度以指导优化。我们还构建了一个执行验证的数据合成流水线,生成多样化高质量训练数据,与推理阶段优化形成互补。在IWR-Bench评测中,RILA在各类基础模型上均同时提升交互与视觉表现。尤其使用该训练流程后,紧凑的Qwen3.5-9B模型准确率从40.40%提升至57.52%,超过参数量更大的单次生成器,包括1万亿参数的Kimi-K2.6(55.61%)和专有GPT-5.5(55.74%)。

原文摘要 · Abstract (English)

Multimodal large language models have achieved remarkable progress in front-end web development, generating interactive webpages from multimodal references such as screenshots and interaction videos. However, existing work largely emphasizes visual metrics such as aesthetics and layout similarity, while overlooking the more critical validation of interactive functionality. We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback. RILA introduces an Action Interaction Verification (AIV) module that replays the reference interaction trajectory on the generated webpage to collect grounded execution-aware observations, and an Execution-aware Rendering Score (ERS) that jointly measures interaction correctness and visual fidelity to guide iterative optimization. We further build an execution-verified data synthesis pipeline that produces diverse, high-quality training data, offering gains complementary to inference-time optimization. On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models. Notably, with our training pipeline, RILA lifts the compact Qwen3.5-9B backbone from 40.40% to 57.52%, surpassing far larger one-shot generators, including the 1T-parameter Kimi-K2.6 (55.61%) and the proprietary GPT-5.5 (55.74%).

网页生成交互验证执行驱动大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。