评测大模型生成交互界面在多轮迭代中的持续维护能力
EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants
- 构建多轮界面维护基准，含150个五轮任务共750轮
- 最强模型仅完成37.3%任务，工具类任务成功率降至52.4%
- 揭示界面演化中状态同步与外部依赖的深层挑战
大型语言模型可生成交互式网页界面，但可靠的生成式用户界面需在用户需求演变时保持可执行的界面实体。本文提出EvoGenUI-Bench，一个用于多轮界面维护的基准测试，包含150个五轮任务、共750轮，覆盖信息展示、可执行交互和工具依赖的外部状态三种场景。通过浏览器执行生成的界面，利用截图、源码与DOM证据、动作轨迹及运行时日志进行评估。除轮次级和回合级成功率外，还引入相邻轮次通过率（Adjacent Pass Retention, APR）衡量跨轮保持能力。在8个模型中，最强模型的单轮通过率为74.9%，但仅完成37.3%的五轮任务；在工具依赖任务中，APR进一步下降至52.4%。诊断分析显示，展示类失败主要源于信息架构问题，交互类失败集中于衍生状态传播与可用性绑定，工具类失败还涉及外部状态对齐与需求拆解。结果表明，生成式界面评估应从孤立输出判断转向检验界面行为、衍生状态、外部状态与助手声明在整个演化过程中的同步性。
原文摘要 · Abstract (English)
Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。