构建首个前端开发对话生成基准，支持多模态反馈评测。
FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback
- 设计多轮对话+图文双输入的前端代码生成任务
- 发现模型严重遗忘历史功能，视觉理解能力普遍不足
- 提出自检式框架，显著减少遗忘并提升性能
我们提出 FronTalk，一个面向前端代码生成的基准，首次研究对话式代码生成中多模态反馈的交互动态。在前端开发中，草图、原型和带注释截图等视觉素材对传达设计意图至关重要，但其在多轮代码生成中的作用尚未被系统探索。为此，我们聚焦前端开发任务，收集了来自新闻、金融、艺术等多个领域真实网站的100组多轮对话，每轮包含文本指令与对应视觉指令，均表达相同用户意图。为全面评估模型表现，我们提出基于代理的评估框架，利用网页代理模拟用户行为，衡量功能正确性与用户体验。对20个模型的评估揭示两个关键挑战：（1）显著的遗忘问题，模型会覆盖之前实现的功能导致任务失败；（2）对视觉反馈的持续理解困难，尤其是开源视觉-语言模型（VLMs）。我们提出强基线方法 AceCoder，通过自主网页代理对每条历史指令的实现进行批判性检查，将遗忘率几乎降至零，性能提升最高达9.3%（56.0% → 65.3%）。本工作旨在为未来前端开发及多轮、多模态代码生成的交互机制研究提供坚实基础。代码与数据已开源。
原文摘要 · Abstract (English)
We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback. In front-end development, visual artifacts such as sketches, mockups and annotated creenshots are essential for conveying design intent, yet their role in multi-turn code generation remains largely unexplored. To address this gap, we focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues derived from real-world websites across diverse domains such as news, finance, and art. Each turn features both a textual instruction and an equivalent visual instruction, each representing the same user intent. To comprehensively evaluate model performance, we propose a novel agent-based evaluation framework leveraging a web agent to simulate users and explore the website, and thus measuring both functional correctness and user experience. Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature: (1) a significant forgetting issue where models overwrite previously implemented features, resulting in task failures, and (2) a persistent challenge in interpreting visual feedback, especially for open-source vision-language models (VLMs). We propose a strong baseline to tackle the forgetting issue with AceCoder, a method that critiques the implementation of every past instruction using an autonomous web agent. This approach significantly reduces forgetting to nearly zero and improves the performance by up to 9.3% (56.0% to 65.3%). Overall, we aim to provide a solid foundation for future research in front-end development and the general interaction dynamics of multi-turn, multi-modal code generation. Code and data are released at https://github.com/shirley-wu/frontalk
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。