让大模型通过示范学习操作手机应用，还能解释自己的行为。
Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration
- 用示范视频+代码生成+界面映射三模块训练可解释的智能助手。
- 在5个应用上任务成功率高，能泛化到未见过的场景。
- 适合想开发可解释自动化工具的研究者或工程师。
随着移动应用复杂度提升，自主进行移动应用交互变得愈发重要。如何开发能有效导航并操作移动应用的智能体仍是重大挑战。本文提出一种可解释的行为克隆大语言模型代理（EBC-LLMAgent），结合大语言模型（LLMs）与基于示范的学习，构建用于自主移动应用交互的智能且可解释的代理。EBC-LLMAgent包含三个核心模块：示范编码、代码生成和界面映射，协同完成用户示范捕捉、可执行代码生成及代码与界面元素间的精确对应。我们引入行为克隆链融合技术以增强代理的泛化能力。在五个来自不同领域的流行移动应用上的大量实验表明，EBC-LLMAgent在任务完成率、对未见场景的高效泛化以及生成有意义解释方面表现优异。
原文摘要 · Abstract (English)
Autonomous mobile app interaction has become increasingly important with growing complexity of mobile applications. Developing intelligent agents that can effectively navigate and interact with mobile apps remains a significant challenge. In this paper, we propose an Explainable Behavior Cloning LLM Agent (EBC-LLMAgent), a novel approach that combines large language models (LLMs) with behavior cloning by learning demonstrations to create intelligent and explainable agents for autonomous mobile app interaction. EBC-LLMAgent consists of three core modules: Demonstration Encoding, Code Generation, and UI Mapping, which work synergistically to capture user demonstrations, generate executable codes, and establish accurate correspondence between code and UI elements. We introduce the Behavior Cloning Chain Fusion technique to enhance the generalization capabilities of the agent. Extensive experiments on five popular mobile applications from diverse domains demonstrate the superior performance of EBC-LLMAgent, achieving high success rates in task completion, efficient generalization to unseen scenarios, and the generation of meaningful explanations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。