优化大模型导航界面的表示,让智能体更省 token 又不丢功能。
From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents
- 用领域语言限制代码空间,结合强化学习迭代优化。
- 在多个平台测试中减少48.7%~55.8%的token消耗。
- 轻量插件形式部署,适合现有大模型智能体使用。
尽管大语言模型(LLM)代理在自动化界面导航方面展现出巨大潜力,如自动化测试与AI助手,但其效率问题长期被忽视。我们的研究发现,低效的界面表示是性能瓶颈。然而,将界面表示优化建模为自动生成转换程序的任务,面临两大挑战:一是缺乏布尔验证器,难以准确判断语义正确性,制约了令牌效率与完整性协同优化;二是需处理大规模复杂界面树并生成长且组合式的转换程序,导致搜索空间庞大且易出错。为此,我们提出UIFormer,首个通过结构化分解与约束优化实现自动界面转换程序合成的框架。首先,采用领域特定语言(DSL)限定程序空间,捕捉界面特有操作;其次,利用基于大模型的迭代精炼机制,结合正确性与效率奖励,引导实现效率与完整性协同优化。UIFormer作为轻量级插件,可无缝集成至现有LLM代理,对核心逻辑修改极少。在覆盖Android与Web平台的三个界面导航基准上,使用五种大模型进行评估,结果显示其在保持或提升代理性能的同时,实现48.7%至55.8%的令牌减少,且运行开销极小。微信真实场景部署进一步验证了其实际价值。
原文摘要 · Abstract (English)
While Large Language Model (LLM) agents show great potential for automated UI navigation such as automated UI testing and AI assistants, their efficiency has been largely overlooked. Our motivating study reveals that inefficient UI representation creates a critical performance bottleneck. However, UI representation optimization, formulated as the task of automatically generating programs that transform UI representations, faces two unique challenges. First, the lack of Boolean oracles, which traditional program synthesis uses to decisively validate semantic correctness, poses a fundamental challenge to co-optimization of token efficiency and completeness. Second, the need to process large, complex UI trees as input while generating long, compositional transformation programs, making the search space vast and error-prone. Toward addressing the preceding limitations, we present UIFormer, the first automated optimization framework that synthesizes UI transformation programs by conducting constraint-based optimization with structured decomposition of the complex synthesis task. First, UIFormer restricts the program space using a domain-specific language (DSL) that captures UI-specific operations. Second, UIFormer conducts LLM-based iterative refinement with correctness and efficiency rewards, providing guidance for achieving the efficiency-completeness co-optimization. UIFormer operates as a lightweight plugin that applies transformation programs for seamless integration with existing LLM agents, requiring minimal modifications to their core logic. Evaluations across three UI navigation benchmarks spanning Android and Web platforms with five LLMs demonstrate that UIFormer achieves 48.7% to 55.8% token reduction with minimal runtime overhead while maintaining or improving agent performance. Real-world industry deployment at WeChat further validates the practical impact of UIFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。