arXiv:2511.15567cs.CVcs.CL2025-11被引 8

让智能代理当裁判,自动设计更高效的界面。

Computer-Use Agents as Judges for Generative User Interface

  • 用代码模型生成界面,代理充当裁判评估功能
  • 52个应用、1560个任务,任务可执行性由程序验证
  • 代理反馈转为可视化指引,适合自动化设计研究者

计算机使用代理(CUA)正日益具备通过图形用户界面(GUI)自主操作数字环境的能力。然而,多数GUI仍以人类为中心,强调美观与易用性,迫使代理采取不必要的行为。与此同时,面向代码的语言模型(Coder)快速发展,推动了自动GUI设计的变革。本文提出:能否让CUA作为裁判协助Coder进行自动界面设计?为此,我们构建了AUI-Gym基准,涵盖52个跨领域应用,通过语言模型合成1560个模拟真实场景的任务,并开发验证器程序化检查任务在环境中是否可执行。在此基础上,提出「编码器-代理协同」框架:编码器作为设计师生成与修改网页,代理作为裁判评估功能并优化设计。成功标准不看外观,而看任务可解性和代理导航成功率。为将代理反馈转化为可操作指导,我们设计了代理仪表盘,将多步导航历史压缩为简洁视觉摘要,支持迭代重设计。该框架推动界面设计向代理原生效率与可靠性演进。工作迈出从被动使用到主动参与数字环境的重要一步。代码与数据集已开源。

原文摘要 · Abstract (English)

Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans--prioritizing aesthetics and usability--forcing agents to adopt human-oriented behaviors that are unnecessary for efficient task execution. At the same time, rapid advances in coding-oriented language models (Coder) have transformed automatic GUI design. This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design? To investigate, we introduce AUI-Gym, a benchmark for Automatic GUI development spanning 52 applications across diverse domains. Using language models, we synthesize 1560 tasks that simulate real-world scenarios. To ensure task reliability, we further develop a verifier that programmatically checks whether each task is executable within its environment. Building on this, we propose a Coder-CUA in Collaboration framework: the Coder acts as Designer, generating and revising websites, while the CUA serves as Judge, evaluating functionality and refining designs. Success is measured not by visual appearance, but by task solvability and CUA navigation success rate. To turn CUA feedback into usable guidance, we design a CUA Dashboard that compresses multi-step navigation histories into concise visual summaries, offering interpretable guidance for iterative redesign. By positioning agents as both designers and judges, our framework shifts interface design toward agent-native efficiency and reliability. Our work takes a step toward shifting agents from passive use toward active participation in digital environments. Our code and dataset are available at https://github.com/showlab/AUI.

自动界面智能代理人机协作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。