arXiv:2510.15455cs.CL2025-10NeurIPS被引 8

通过云端与本地大模型协作,减少手机自动化任务中的界面信息泄露。

CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs

  • 基于界面布局分块,只上传相关区域信息
  • 云与本地模型协同规划决策,界面暴露降低55.6%
  • 适合注重隐私的移动自动化应用开发者

移动端智能体依赖大语言模型(LLMs)在手机用户界面(UI)上规划和执行任务。云端大模型虽准确率高,但每步需上传完整界面状态,导致不必要的信息暴露;本地大模型避免上传,却因能力受限而成功率较低。本文提出CORE框架,融合云端与本地大模型优势,在保持任务准确性的同时显著减少界面信息暴露。CORE包含三个核心组件:(1) 布局感知的块划分,根据XML层级结构分组语义相关的界面元素;(2) 协同规划,由本地与云端大模型共同识别当前子任务;(3) 协同决策,本地模型对相关界面块排序,云端模型在排名靠前的块内选择具体元素。此外,引入多轮累积机制以缓解本地误判或上下文限制。跨多种移动应用与任务的实验表明,CORE可将界面暴露降低至最多55.6%,任务成功率略低于纯云端方案,有效缓解了向云端泄露无关隐私的问题。代码已开源:https://github.com/Entropy-Fighter/CORE。

原文摘要 · Abstract (English)

Mobile agents rely on Large Language Models (LLMs) to plan and execute tasks on smartphone user interfaces (UIs). While cloud-based LLMs achieve high task accuracy, they require uploading the full UI state at every step, exposing unnecessary and often irrelevant information. In contrast, local LLMs avoid UI uploads but suffer from limited capacity, resulting in lower task success rates. We propose $\textbf{CORE}$, a $\textbf{CO}$llaborative framework that combines the strengths of cloud and local LLMs to $\textbf{R}$educe UI $\textbf{E}$xposure, while maintaining task accuracy for mobile agents. CORE comprises three key components: (1) $\textbf{Layout-aware block partitioning}$, which groups semantically related UI elements based on the XML screen hierarchy; (2) $\textbf{Co-planning}$, where local and cloud LLMs collaboratively identify the current sub-task; and (3) $\textbf{Co-decision-making}$, where the local LLM ranks relevant UI blocks, and the cloud LLM selects specific UI elements within the top-ranked block. CORE further introduces a multi-round accumulation mechanism to mitigate local misjudgment or limited context. Experiments across diverse mobile apps and tasks show that CORE reduces UI exposure by up to 55.6% while maintaining task success rates slightly below cloud-only agents, effectively mitigating unnecessary privacy exposure to the cloud. The code is available at https://github.com/Entropy-Fighter/CORE.

移动智能体隐私保护大模型协作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。