arXiv:2506.17697cs.AI2025-06被引 1

让手机应用智能体学会理解操作语义,提升鲁棒性和泛化能力。

Beyond Syntax: Action Semantics Learning for App Agents

  • 基于界面状态转移定义操作语义,不依赖精确指令匹配
  • 在多个离线与在线基准上准确率显著优于现有方法
  • 适用于小模型微调,降低算力成本且增强对抗分布外情况的能力

大型语言模型(LLM)推动了应用智能体的发展,使其能通过点击、滚动等操作理解用户意图并操控手机应用。尽管基于提示的方案表现良好,但存在高昂计算成本和对外部API的依赖问题。微调小型开源LLM可缓解此问题,但现有监督微调方法采用语法学习范式,要求完全复现真实操作字符串,导致分布外(OOD)场景下脆弱。为此,本文提出动作语义学习(ASL),其学习目标是捕捉真实动作的语义。受编程语言理论启发,将应用智能体的动作语义定义为动作引起的界面状态转移。基于此,ASL引入一种新型语义估计器(SEE),通过计算语义相似度来训练智能体生成与真实动作语义一致但语法不同的操作。SEE模块灵活,适用于监督与强化微调。理论分析证明ASL在应对OOD问题上优于传统语法学习。大量实验表明,ASL在多个离线与在线基准上显著提升应用智能体的准确率与泛化能力。

原文摘要 · Abstract (English)

The recent development of Large Language Models (LLMs) enables the rise of App agents that interpret user intent and operate smartphone Apps through actions such as clicking and scrolling. While prompt-based solutions with proprietary LLM APIs show promising ability, they incur heavy compute costs and external API dependency. Fine-tuning smaller open-source LLMs solves these limitations. However, current supervised fine-tuning methods use a syntax learning paradigm that forces agents to reproduce exactly the ground truth action strings, leading to out-of-distribution (OOD) vulnerability. To fill this gap, we propose Action Semantics Learning (ASL), a novel learning framework, where the learning objective is capturing the semantics of the ground truth actions. Specifically, inspired by the programming language theory, we define the action semantics for App agents as the state transition induced by the action in the user interface. Building on this insight, ASL employs a novel SEmantic Estimator~(SEE) to compute a semantic similarity to train the App agents in generating actions aligned with the semantics of ground truth actions, even when their syntactic forms differ. SEE is a flexible module that can be applied in both supervised and reinforcement fine-tuning paradigms. To support the effectiveness of ASL, we theoretically demonstrate the superior robustness of ASL for the OOD problem compared with the existing syntax learning paradigm. Extensive experiments across multiple offline and online benchmarks demonstrate that ASL significantly improves the accuracy and generalisation of App agents compared to existing methods.

应用智能体语义学习大模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。