arXiv:2606.12666cs.CRcs.AI2026-06

CAPED让手机界面截图只暴露任务所需内容,保护无关隐私。

CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents

论文配图:CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents
图 1 · 摘自论文原文
  • 根据任务需求和屏幕上下文,选择性暴露必要信息
  • 在28项任务中将隐私泄露率从0.766降至0.268
  • 适合需远程操控手机的智能助手场景

基于截图的移动GUI代理能像人类一样操作手机应用,但也使每次屏幕观察都成为隐私边界。正常任务执行中,截图可能暴露联系人、消息、照片、文件、推荐内容、健康线索等与用户请求无关的敏感信息,称为意外视觉隐私泄露。现有防护手段难以应对:文本匿名化遗漏大量视觉与推断线索,通用隐私遮蔽又会移除代理完成任务所需的证据。本文提出CAPED,一种面向移动GUI代理的上下文感知预上传隐私防护层。该系统部署于手机端,在截图发送至远程多模态代理前,提取任务需求,利用屏幕上下文作为隐私先验,解析可见UI元素,仅保留当前任务必需内容,对非必要隐私内容进行遮蔽。我们在AndroidWorld上评估其任务泛用性,并通过28项任务的种子隐私泄露测试衡量轨迹级意外泄露。结果显示,完整版CAPED将成功条件加权种子泄露率从原始截图的0.766降至0.268,同时保持高任务完成率。更广泛的AndroidWorld实验表明仍存在原型级任务效用损耗,但结果证明任务驱动的选择性暴露可有效减少截图上传前的意外视觉泄露。

原文摘要 · Abstract (English)

Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results show that task-driven selective exposure can reduce incidental visual leakage before screenshots are released to a remote GUI agent.

隐私保护GUI代理移动端视觉隐私

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。